2023-07-14T17:43:26.776 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e 2023-07-14T17:43:26.777 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990 --name pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi --verbose -- /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/orig.config.yaml 2023-07-14T17:43:26.832 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T17:43:26.950 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990 branch: wip-pdonnell-testing-20230714.134335 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} email: pdonnell@redhat.com first_in_suite: false job_id: '7337990' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230714.134335 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.dir_split install: ceph: flavor: default sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec kclient: client.0: mount_subvol_num: 1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230714.134335 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5179 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec sleep_before_teardown: 0 subset: 285/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230714.134335 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi115.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi150.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs _nogroup random 0.01 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: aa6a80e6cc655fff48452a7eca02070e61d074fc timestamp: 2023-07-14_16:44:47 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-14T17:43:26.951 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa; will attempt to use it 2023-07-14T17:43:26.952 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa/tasks 2023-07-14T17:43:26.952 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-14T17:43:26.954 INFO:teuthology.task.internal:Checking packages... 2023-07-14T17:43:26.982 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '81c9387d2e7576da065747ec9c16d1c7d69016ec' 2023-07-14T17:43:26.982 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-14T17:43:26.983 INFO:teuthology.packaging:ref: None 2023-07-14T17:43:26.983 INFO:teuthology.packaging:tag: None 2023-07-14T17:43:26.983 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230714.134335 2023-07-14T17:43:26.983 INFO:teuthology.packaging:sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:43:26.984 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230714.134335 2023-07-14T17:43:27.375 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4940.g81c9387d 2023-07-14T17:43:27.375 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-14T17:43:27.386 INFO:teuthology.task.internal:no buildpackages task found 2023-07-14T17:43:27.387 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-14T17:43:27.414 INFO:teuthology.task.internal:Saving configuration 2023-07-14T17:43:27.438 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-14T17:43:27.444 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T17:43:27.469 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:35:29.870021', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T17:43:27.493 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:35:29.868687', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T17:43:27.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:35:29.867096', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T17:43:27.516 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-14T17:43:27.523 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-14T17:43:27.524 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-14T17:43:27.524 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-14T17:43:27.524 INFO:teuthology.run_tasks:Running task console_log... 2023-07-14T17:43:27.651 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc416d813a0>, signals=[15]) 2023-07-14T17:43:27.653 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-14T17:43:27.661 INFO:teuthology.task.internal:Opening connections... 2023-07-14T17:43:27.662 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2023-07-14T17:43:27.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:43:27.734 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2023-07-14T17:43:27.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:43:27.803 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2023-07-14T17:43:27.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:43:27.872 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-14T17:43:27.879 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2023-07-14T17:43:27.901 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2023-07-14T17:43:27.901 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2023-07-14T17:43:27.961 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T17:43:27.961 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T17:43:27.962 INFO:teuthology.orchestra.run.smithi115.stdout:ID="rhel" 2023-07-14T17:43:27.962 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="fedora" 2023-07-14T17:43:27.962 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8.6" 2023-07-14T17:43:27.962 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8" 2023-07-14T17:43:27.962 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T17:43:27.962 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2023-07-14T17:43:27.962 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T17:43:27.963 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T17:43:27.963 INFO:teuthology.orchestra.run.smithi115.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T17:43:27.963 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T17:43:27.963 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T17:43:27.963 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T17:43:27.963 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T17:43:27.963 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T17:43:27.964 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T17:43:27.964 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2023-07-14T17:43:27.985 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2023-07-14T17:43:28.006 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-07-14T17:43:28.007 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2023-07-14T17:43:28.066 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T17:43:28.067 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T17:43:28.067 INFO:teuthology.orchestra.run.smithi150.stdout:ID="rhel" 2023-07-14T17:43:28.067 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="fedora" 2023-07-14T17:43:28.067 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8.6" 2023-07-14T17:43:28.068 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2023-07-14T17:43:28.068 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T17:43:28.068 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-07-14T17:43:28.068 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T17:43:28.069 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T17:43:28.069 INFO:teuthology.orchestra.run.smithi150.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T17:43:28.069 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T17:43:28.069 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T17:43:28.070 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T17:43:28.070 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T17:43:28.070 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T17:43:28.071 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T17:43:28.071 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-07-14T17:43:28.093 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2023-07-14T17:43:28.113 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2023-07-14T17:43:28.114 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2023-07-14T17:43:28.172 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T17:43:28.172 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T17:43:28.173 INFO:teuthology.orchestra.run.smithi187.stdout:ID="rhel" 2023-07-14T17:43:28.173 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="fedora" 2023-07-14T17:43:28.173 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8.6" 2023-07-14T17:43:28.173 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2023-07-14T17:43:28.173 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T17:43:28.173 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2023-07-14T17:43:28.173 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T17:43:28.174 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T17:43:28.174 INFO:teuthology.orchestra.run.smithi187.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T17:43:28.174 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T17:43:28.174 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T17:43:28.174 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T17:43:28.174 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T17:43:28.174 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T17:43:28.174 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T17:43:28.175 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2023-07-14T17:43:28.194 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-14T17:43:28.204 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-14T17:43:28.254 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-14T17:43:28.254 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2023-07-14T17:43:28.257 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2023-07-14T17:43:28.259 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2023-07-14T17:43:28.277 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-14T17:43:28.322 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-14T17:43:28.322 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2023-07-14T17:43:28.325 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2023-07-14T17:43:28.327 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2023-07-14T17:43:28.357 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-14T17:43:28.441 INFO:teuthology.run_tasks:Running task kernel... 2023-07-14T17:43:28.459 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-14T17:43:28.459 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-14T17:43:28.460 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T17:43:28.461 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T17:43:28.461 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T17:43:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:43:28.479 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2023-07-14T17:43:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:43:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:43:28.481 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-07-14T17:43:28.481 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2023-07-14T17:43:28.537 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:28.538 INFO:teuthology.task.kernel:Running kernel on smithi187: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:28.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2023-07-14T17:43:28.540 INFO:teuthology.orchestra.run.smithi150.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:28.540 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:28.541 INFO:teuthology.task.kernel:Running kernel on smithi150: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:28.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2023-07-14T17:43:28.541 INFO:teuthology.task.kernel:Running kernel on smithi115: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:28.541 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2023-07-14T17:43:29.726 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-07-14T17:43:29.726 INFO:teuthology.orchestra.run.smithi187.stdout:Unable to read consumer identity 2023-07-14T17:43:29.751 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-07-14T17:43:29.752 INFO:teuthology.orchestra.run.smithi115.stdout:Unable to read consumer identity 2023-07-14T17:43:29.762 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-07-14T17:43:29.763 INFO:teuthology.orchestra.run.smithi150.stdout:Unable to read consumer identity 2023-07-14T17:43:31.274 INFO:teuthology.orchestra.run.smithi187.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-07-14T17:43:31.333 INFO:teuthology.orchestra.run.smithi150.stdout:Copr repo for python3-asyncssh owned by ceph 9.6 kB/s | 3.5 kB 00:00 2023-07-14T17:43:31.416 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-07-14T17:43:34.591 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 16 MB 00:03 2023-07-14T17:43:34.658 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 16 MB 00:03 2023-07-14T17:43:34.666 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 4.8 MB/s | 16 MB 00:03 2023-07-14T17:43:38.158 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 413 kB/s | 24 kB 00:00 2023-07-14T17:43:38.159 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 560 kB/s | 24 kB 00:00 2023-07-14T17:43:38.209 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 436 kB/s | 24 kB 00:00 2023-07-14T17:43:38.802 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T17:43:38.862 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-07-14T17:43:38.863 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-07-14T17:43:38.863 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-07-14T17:43:38.869 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T17:43:38.877 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T17:43:38.922 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-07-14T17:43:38.923 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-07-14T17:43:38.924 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-07-14T17:43:38.924 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-07-14T17:43:38.924 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-07-14T17:43:38.925 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-07-14T17:43:39.029 DEBUG:teuthology.orchestra.run.smithi187:> echo no | sudo yum reinstall kernel || true 2023-07-14T17:43:39.039 DEBUG:teuthology.orchestra.run.smithi115:> echo no | sudo yum reinstall kernel || true 2023-07-14T17:43:39.041 DEBUG:teuthology.orchestra.run.smithi150:> echo no | sudo yum reinstall kernel || true 2023-07-14T17:43:39.373 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-07-14T17:43:39.373 INFO:teuthology.orchestra.run.smithi150.stdout:Unable to read consumer identity 2023-07-14T17:43:39.402 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-07-14T17:43:39.402 INFO:teuthology.orchestra.run.smithi187.stdout:Unable to read consumer identity 2023-07-14T17:43:39.411 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-07-14T17:43:39.411 INFO:teuthology.orchestra.run.smithi115.stdout:Unable to read consumer identity 2023-07-14T17:43:39.639 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:01 ago on Fri 14 Jul 2023 05:43:38 PM UTC. 2023-07-14T17:43:39.669 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:01 ago on Fri 14 Jul 2023 05:43:38 PM UTC. 2023-07-14T17:43:39.677 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:01 ago on Fri 14 Jul 2023 05:43:38 PM UTC. 2023-07-14T17:43:39.679 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:43:39.711 INFO:teuthology.orchestra.run.smithi187.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:43:39.718 INFO:teuthology.orchestra.run.smithi115.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:43:39.754 INFO:teuthology.orchestra.run.smithi187.stderr:Error: No packages marked for reinstall. 2023-07-14T17:43:39.758 INFO:teuthology.orchestra.run.smithi115.stderr:Error: No packages marked for reinstall. 2023-07-14T17:43:39.769 INFO:teuthology.orchestra.run.smithi150.stderr:Error: No packages marked for reinstall. 2023-07-14T17:43:39.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum reinstall -y kernel || true 2023-07-14T17:43:39.805 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum reinstall -y kernel || true 2023-07-14T17:43:39.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum reinstall -y kernel || true 2023-07-14T17:43:40.132 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-07-14T17:43:40.133 INFO:teuthology.orchestra.run.smithi187.stdout:Unable to read consumer identity 2023-07-14T17:43:40.134 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-07-14T17:43:40.134 INFO:teuthology.orchestra.run.smithi115.stdout:Unable to read consumer identity 2023-07-14T17:43:40.145 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-07-14T17:43:40.146 INFO:teuthology.orchestra.run.smithi150.stdout:Unable to read consumer identity 2023-07-14T17:43:40.392 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:43:38 PM UTC. 2023-07-14T17:43:40.395 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:43:38 PM UTC. 2023-07-14T17:43:40.400 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:43:38 PM UTC. 2023-07-14T17:43:40.431 INFO:teuthology.orchestra.run.smithi115.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:43:40.435 INFO:teuthology.orchestra.run.smithi187.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:43:40.442 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:43:40.463 INFO:teuthology.orchestra.run.smithi115.stderr:Error: No packages marked for reinstall. 2023-07-14T17:43:40.469 INFO:teuthology.orchestra.run.smithi187.stderr:Error: No packages marked for reinstall. 2023-07-14T17:43:40.482 INFO:teuthology.orchestra.run.smithi150.stderr:Error: No packages marked for reinstall. 2023-07-14T17:43:40.514 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T17:43:40.516 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T17:43:40.528 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T17:43:40.593 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:40.594 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:40.594 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T17:43:40.594 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-14T17:43:40.595 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T17:43:40.596 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:40.597 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:40.598 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T17:43:40.598 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-07-14T17:43:40.598 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T17:43:40.608 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:40.608 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:43:40.608 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T17:43:40.609 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-14T17:43:40.609 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T17:43:40.626 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2023-07-14T17:43:40.629 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2023-07-14T17:43:40.637 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2023-07-14T17:43:40.641 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:40.642 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:40.651 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:40.651 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-14T17:43:40.660 INFO:teuthology.task.internal:Creating test directory... 2023-07-14T17:43:40.660 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T17:43:40.683 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T17:43:40.693 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T17:43:40.709 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-14T17:43:40.719 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-14T17:43:40.805 INFO:teuthology.task.internal:Creating archive directory... 2023-07-14T17:43:40.806 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T17:43:40.808 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T17:43:40.810 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T17:43:40.841 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-14T17:43:40.847 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-14T17:43:40.848 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T17:43:40.866 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T17:43:40.869 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T17:43:40.896 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:43:40.900 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:43:40.905 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:43:40.906 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:43:40.910 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:43:40.915 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:43:40.916 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-14T17:43:40.926 INFO:teuthology.task.internal:Configuring sudo... 2023-07-14T17:43:40.926 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T17:43:40.950 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T17:43:40.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T17:43:40.982 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-14T17:43:40.991 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-14T17:43:40.991 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T17:43:41.017 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T17:43:41.023 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T17:43:41.039 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:43:51.128 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:43:51.171 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:43:56.262 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:43:56.293 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T17:43:56.294 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T17:43:56.385 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:44:01.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:44:01.485 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:44:06.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:44:06.610 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T17:44:06.610 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T17:44:06.692 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:44:06.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:44:06.797 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:44:11.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:44:11.919 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T17:44:11.920 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T17:44:11.990 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2023-07-14T17:44:11.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2023-07-14T17:44:11.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2023-07-14T17:44:12.058 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:44:12.068 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:44:12.096 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:44:12.391 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-14T17:44:12.399 INFO:teuthology.task.internal:Starting timer... 2023-07-14T17:44:12.399 INFO:teuthology.run_tasks:Running task pcp... 2023-07-14T17:44:12.491 INFO:teuthology.run_tasks:Running task selinux... 2023-07-14T17:44:12.530 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-14T17:44:12.531 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2023-07-14T17:44:12.586 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2023-07-14T17:44:12.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2023-07-14T17:44:12.641 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2023-07-14T17:44:12.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2023-07-14T17:44:12.703 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2023-07-14T17:44:12.705 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-14T17:44:12.705 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2023-07-14T17:44:12.728 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2023-07-14T17:44:12.729 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2023-07-14T17:44:12.749 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2023-07-14T17:44:12.750 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2023-07-14T17:44:12.774 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2023-07-14T17:44:12.775 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi115.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2023-07-14T17:44:12.775 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:44:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:44:12.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:44:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:44:12.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:44:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:44:12.859 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-14T17:44:12.859 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2023-07-14T17:44:12.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2023-07-14T17:44:12.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2023-07-14T17:44:12.939 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-14T17:44:12.949 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-07-14T17:44:12.950 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-14T17:44:12.970 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-14T17:44:12.972 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi115.front.sepia.ceph.com,smithi150.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-14T17:58:19.216 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi115.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2023-07-14T17:58:19.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-07-14T17:58:19.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:58:24.333 DEBUG:teuthology.orchestra.run.smithi115:> true 2023-07-14T17:58:24.403 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-07-14T17:58:24.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-07-14T17:58:24.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:58:29.479 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-07-14T17:58:29.549 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-07-14T17:58:29.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-07-14T17:58:29.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:58:34.622 DEBUG:teuthology.orchestra.run.smithi187:> true 2023-07-14T17:58:34.697 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-07-14T17:58:34.698 INFO:teuthology.run_tasks:Running task clock... 2023-07-14T17:58:34.708 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-14T17:58:34.709 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T17:58:34.710 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T17:58:34.712 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T17:58:34.712 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T17:58:34.714 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T17:58:34.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T17:58:34.745 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T17:58:34.749 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T17:58:34.762 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T17:58:34.765 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T17:58:34.774 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T17:58:34.789 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T17:58:34.831 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2023-07-14T17:58:34.843 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2023-07-14T17:58:34.843 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2023-07-14T17:58:34.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2023-07-14T17:58:34.857 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2023-07-14T17:58:34.857 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T17:58:34.869 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2023-07-14T17:58:34.871 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T17:58:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T17:58:34.883 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T17:58:34.887 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T17:58:34.898 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T17:58:34.943 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2023-07-14T17:58:34.946 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T17:58:34.946 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2023-07-14T17:58:34.946 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.946 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.947 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.947 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.960 INFO:teuthology.orchestra.run.smithi150.stderr:bash: ntpq: command not found 2023-07-14T17:58:34.964 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T17:58:34.965 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-07-14T17:58:34.965 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.965 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.965 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.965 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.966 INFO:teuthology.orchestra.run.smithi187.stderr:bash: ntpq: command not found 2023-07-14T17:58:34.970 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T17:58:34.970 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2023-07-14T17:58:34.970 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.970 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.970 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.970 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:58:34.971 INFO:teuthology.run_tasks:Running task pexec... 2023-07-14T17:58:34.981 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-14T17:58:34.982 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi115.front.sepia.ceph.com 2023-07-14T17:58:34.982 DEBUG:teuthology.orchestra.run.smithi115:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T17:58:34.984 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi150.front.sepia.ceph.com 2023-07-14T17:58:34.984 DEBUG:teuthology.orchestra.run.smithi150:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T17:58:34.984 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi187.front.sepia.ceph.com 2023-07-14T17:58:34.985 DEBUG:teuthology.orchestra.run.smithi187:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T17:58:35.357 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-07-14T17:58:35.375 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-07-14T17:58:35.384 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-07-14T17:58:36.845 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:00 ago on Fri 14 Jul 2023 05:56:36 PM UTC. 2023-07-14T17:58:37.071 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:06:30 ago on Fri 14 Jul 2023 05:52:06 PM UTC. 2023-07-14T17:58:37.171 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:06:35 ago on Fri 14 Jul 2023 05:52:01 PM UTC. 2023-07-14T17:58:38.625 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-07-14T17:58:38.626 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-07-14T17:58:38.627 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-07-14T17:58:38.627 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-07-14T17:58:38.627 INFO:teuthology.orchestra.run.smithi150.stdout:Resetting modules: 2023-07-14T17:58:38.627 INFO:teuthology.orchestra.run.smithi150.stdout: container-tools 2023-07-14T17:58:38.627 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T17:58:38.627 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-07-14T17:58:38.627 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-07-14T17:58:38.628 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T17:58:38.780 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-07-14T17:58:38.781 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-07-14T17:58:38.781 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2023-07-14T17:58:38.781 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-07-14T17:58:38.781 INFO:teuthology.orchestra.run.smithi187.stdout:Resetting modules: 2023-07-14T17:58:38.781 INFO:teuthology.orchestra.run.smithi187.stdout: container-tools 2023-07-14T17:58:38.782 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T17:58:38.782 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-07-14T17:58:38.782 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-07-14T17:58:38.782 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T17:58:38.889 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-07-14T17:58:38.890 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-07-14T17:58:38.891 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2023-07-14T17:58:38.891 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-07-14T17:58:38.891 INFO:teuthology.orchestra.run.smithi115.stdout:Resetting modules: 2023-07-14T17:58:38.891 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools 2023-07-14T17:58:38.891 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T17:58:38.891 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-07-14T17:58:38.892 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2023-07-14T17:58:38.892 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T17:58:39.591 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:03 ago on Fri 14 Jul 2023 05:56:36 PM UTC. 2023-07-14T17:58:39.726 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:06:33 ago on Fri 14 Jul 2023 05:52:06 PM UTC. 2023-07-14T17:58:39.824 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:06:38 ago on Fri 14 Jul 2023 05:52:01 PM UTC. 2023-07-14T17:58:46.850 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-07-14T17:58:47.316 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-07-14T17:58:48.139 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-07-14T17:58:48.535 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-07-14T17:58:48.597 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-07-14T17:58:49.000 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-07-14T17:58:49.116 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:12 ago on Fri 14 Jul 2023 05:56:36 PM UTC. 2023-07-14T17:58:50.301 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:06:48 ago on Fri 14 Jul 2023 05:52:01 PM UTC. 2023-07-14T17:58:50.464 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:06:44 ago on Fri 14 Jul 2023 05:52:06 PM UTC. 2023-07-14T17:58:50.844 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-07-14T17:58:50.936 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================================ 2023-07-14T17:58:50.936 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-07-14T17:58:50.937 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================================ 2023-07-14T17:58:50.937 INFO:teuthology.orchestra.run.smithi150.stdout:Installing group/module packages: 2023-07-14T17:58:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T17:58:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T17:58:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T17:58:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T17:58:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T17:58:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T17:58:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T17:58:50.938 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-07-14T17:58:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T17:58:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T17:58:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T17:58:50.938 INFO:teuthology.orchestra.run.smithi150.stdout:Installing module profiles: 2023-07-14T17:58:50.938 INFO:teuthology.orchestra.run.smithi150.stdout: container-tools/common 2023-07-14T17:58:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T17:58:50.939 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling module streams: 2023-07-14T17:58:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: container-tools rhel8 2023-07-14T17:58:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T17:58:50.939 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-07-14T17:58:50.939 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================================ 2023-07-14T17:58:50.939 INFO:teuthology.orchestra.run.smithi150.stdout:Install 10 Packages 2023-07-14T17:58:50.939 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T17:58:50.940 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 18 M 2023-07-14T17:58:50.940 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 65 M 2023-07-14T17:58:50.940 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-07-14T17:58:51.197 INFO:teuthology.orchestra.run.smithi150.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 363 kB/s | 94 kB 00:00 2023-07-14T17:58:51.222 INFO:teuthology.orchestra.run.smithi150.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 88 kB/s | 25 kB 00:00 2023-07-14T17:58:51.247 INFO:teuthology.orchestra.run.smithi150.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 132 kB/s | 41 kB 00:00 2023-07-14T17:58:51.348 INFO:teuthology.orchestra.run.smithi150.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 330 kB/s | 49 kB 00:00 2023-07-14T17:58:51.381 INFO:teuthology.orchestra.run.smithi150.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.2 MB/s | 524 kB 00:00 2023-07-14T17:58:51.481 INFO:teuthology.orchestra.run.smithi150.stdout:(6/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.1 MB/s | 213 kB 00:00 2023-07-14T17:58:51.507 INFO:teuthology.orchestra.run.smithi150.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 941 kB/s | 149 kB 00:00 2023-07-14T17:58:51.640 INFO:teuthology.orchestra.run.smithi150.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 20 MB/s | 8.0 MB 00:00 2023-07-14T17:58:51.724 INFO:teuthology.orchestra.run.smithi150.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 10 MB/s | 2.2 MB 00:00 2023-07-14T17:58:51.841 INFO:teuthology.orchestra.run.smithi150.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 20 MB/s | 7.0 MB 00:00 2023-07-14T17:58:51.841 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:58:51.841 INFO:teuthology.orchestra.run.smithi150.stdout:Total 20 MB/s | 18 MB 00:00 2023-07-14T17:58:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-07-14T17:58:52.042 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-07-14T17:58:52.060 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-07-14T17:58:52.061 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-07-14T17:58:52.099 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================================================ 2023-07-14T17:58:52.099 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-07-14T17:58:52.099 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================================================ 2023-07-14T17:58:52.099 INFO:teuthology.orchestra.run.smithi115.stdout:Installing group/module packages: 2023-07-14T17:58:52.099 INFO:teuthology.orchestra.run.smithi115.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T17:58:52.100 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T17:58:52.100 INFO:teuthology.orchestra.run.smithi115.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T17:58:52.100 INFO:teuthology.orchestra.run.smithi115.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T17:58:52.100 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T17:58:52.100 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T17:58:52.100 INFO:teuthology.orchestra.run.smithi115.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T17:58:52.101 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-07-14T17:58:52.101 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T17:58:52.101 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T17:58:52.101 INFO:teuthology.orchestra.run.smithi115.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T17:58:52.101 INFO:teuthology.orchestra.run.smithi115.stdout:Installing module profiles: 2023-07-14T17:58:52.101 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools/common 2023-07-14T17:58:52.102 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T17:58:52.102 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling module streams: 2023-07-14T17:58:52.102 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools rhel8 2023-07-14T17:58:52.102 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T17:58:52.102 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-07-14T17:58:52.102 INFO:teuthology.orchestra.run.smithi115.stdout:============================================================================================================ 2023-07-14T17:58:52.102 INFO:teuthology.orchestra.run.smithi115.stdout:Install 10 Packages 2023-07-14T17:58:52.103 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T17:58:52.103 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 18 M 2023-07-14T17:58:52.103 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 65 M 2023-07-14T17:58:52.104 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-07-14T17:58:52.235 INFO:teuthology.orchestra.run.smithi115.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 703 kB/s | 94 kB 00:00 2023-07-14T17:58:52.242 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-07-14T17:58:52.246 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-07-14T17:58:52.285 INFO:teuthology.orchestra.run.smithi115.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 135 kB/s | 25 kB 00:00 2023-07-14T17:58:52.302 INFO:teuthology.orchestra.run.smithi115.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 203 kB/s | 41 kB 00:00 2023-07-14T17:58:52.364 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-07-14T17:58:52.436 INFO:teuthology.orchestra.run.smithi115.stdout:(4/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.4 MB/s | 524 kB 00:00 2023-07-14T17:58:52.458 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================================ 2023-07-14T17:58:52.458 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-07-14T17:58:52.458 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================================ 2023-07-14T17:58:52.458 INFO:teuthology.orchestra.run.smithi187.stdout:Installing group/module packages: 2023-07-14T17:58:52.458 INFO:teuthology.orchestra.run.smithi187.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T17:58:52.458 INFO:teuthology.orchestra.run.smithi187.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T17:58:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T17:58:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T17:58:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T17:58:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T17:58:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T17:58:52.459 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-07-14T17:58:52.459 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T17:58:52.460 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T17:58:52.460 INFO:teuthology.orchestra.run.smithi187.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T17:58:52.460 INFO:teuthology.orchestra.run.smithi187.stdout:Installing module profiles: 2023-07-14T17:58:52.460 INFO:teuthology.orchestra.run.smithi187.stdout: container-tools/common 2023-07-14T17:58:52.460 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T17:58:52.460 INFO:teuthology.orchestra.run.smithi187.stdout:Enabling module streams: 2023-07-14T17:58:52.460 INFO:teuthology.orchestra.run.smithi187.stdout: container-tools rhel8 2023-07-14T17:58:52.460 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T17:58:52.461 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-07-14T17:58:52.461 INFO:teuthology.orchestra.run.smithi187.stdout:============================================================================================================ 2023-07-14T17:58:52.461 INFO:teuthology.orchestra.run.smithi187.stdout:Install 10 Packages 2023-07-14T17:58:52.461 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T17:58:52.461 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 18 M 2023-07-14T17:58:52.462 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 65 M 2023-07-14T17:58:52.462 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-07-14T17:58:52.472 INFO:teuthology.orchestra.run.smithi115.stdout:(5/10): udica-0.2.6-4.module+el8.6.0+16771+28df 211 kB/s | 49 kB 00:00 2023-07-14T17:58:52.578 INFO:teuthology.orchestra.run.smithi115.stdout:(6/10): buildah-1.26.2-1.module+el8.6.0+16771+2 29 MB/s | 8.0 MB 00:00 2023-07-14T17:58:52.611 INFO:teuthology.orchestra.run.smithi115.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.5 MB/s | 213 kB 00:00 2023-07-14T17:58:52.616 INFO:teuthology.orchestra.run.smithi187.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 602 kB/s | 94 kB 00:00 2023-07-14T17:58:52.641 INFO:teuthology.orchestra.run.smithi187.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 137 kB/s | 25 kB 00:00 2023-07-14T17:58:52.692 INFO:teuthology.orchestra.run.smithi187.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 176 kB/s | 41 kB 00:00 2023-07-14T17:58:52.768 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-07-14T17:58:52.784 INFO:teuthology.orchestra.run.smithi187.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 295 kB/s | 49 kB 00:00 2023-07-14T17:58:52.820 INFO:teuthology.orchestra.run.smithi115.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 11 MB/s | 2.2 MB 00:00 2023-07-14T17:58:52.826 INFO:teuthology.orchestra.run.smithi187.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.8 MB/s | 524 kB 00:00 2023-07-14T17:58:52.845 INFO:teuthology.orchestra.run.smithi115.stdout:(9/10): python3-podman-4.0.1-1.module+el8.6.0+1 365 kB/s | 149 kB 00:00 2023-07-14T17:58:52.893 INFO:teuthology.orchestra.run.smithi187.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.3 MB/s | 149 kB 00:00 2023-07-14T17:58:52.926 INFO:teuthology.orchestra.run.smithi187.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.1 MB/s | 213 kB 00:00 2023-07-14T17:58:52.937 INFO:teuthology.orchestra.run.smithi115.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 20 MB/s | 7.0 MB 00:00 2023-07-14T17:58:52.937 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:58:52.938 INFO:teuthology.orchestra.run.smithi115.stdout:Total 22 MB/s | 18 MB 00:00 2023-07-14T17:58:52.984 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T17:58:53.051 INFO:teuthology.orchestra.run.smithi187.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 22 MB/s | 8.0 MB 00:00 2023-07-14T17:58:53.137 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-07-14T17:58:53.156 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-07-14T17:58:53.157 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-07-14T17:58:53.158 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:58:53.177 INFO:teuthology.orchestra.run.smithi187.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 8.8 MB/s | 2.2 MB 00:00 2023-07-14T17:58:53.261 INFO:teuthology.orchestra.run.smithi187.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 19 MB/s | 7.0 MB 00:00 2023-07-14T17:58:53.261 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:58:53.262 INFO:teuthology.orchestra.run.smithi187.stdout:Total 23 MB/s | 18 MB 00:00 2023-07-14T17:58:53.338 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-07-14T17:58:53.341 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-07-14T17:58:53.344 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T17:58:53.462 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-07-14T17:58:53.481 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-07-14T17:58:53.481 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-07-14T17:58:53.547 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T17:58:53.662 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-07-14T17:58:53.667 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-07-14T17:58:53.860 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-07-14T17:58:53.911 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T17:58:54.036 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T17:58:54.194 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-07-14T17:58:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:58:54.355 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T17:58:54.427 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T17:58:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:58:54.634 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T17:58:54.666 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T17:58:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T17:58:54.910 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T17:58:55.036 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T17:58:55.252 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T17:58:55.555 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T17:58:55.780 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T17:58:55.793 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T17:58:55.982 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T17:58:56.020 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T17:58:56.159 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:58:56.428 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:58:56.428 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T17:58:56.428 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:58:56.428 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T17:58:56.429 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T17:58:56.429 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T17:58:56.429 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T17:58:56.429 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T17:58:56.429 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T17:58:56.429 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T17:58:56.672 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T17:58:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T17:58:56.904 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T17:58:57.081 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T17:58:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T17:58:57.275 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:58:57.281 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T17:58:57.483 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:58:57.549 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:58:57.549 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T17:58:57.550 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:58:57.550 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T17:58:57.550 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T17:58:57.550 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T17:58:57.550 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T17:58:57.551 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T17:58:57.551 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T17:58:57.551 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T17:58:57.656 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T17:58:57.656 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:21 ago on Fri 14 Jul 2023 05:56:36 PM UTC. 2023-07-14T17:58:57.736 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:58:57.736 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T17:58:57.736 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:58:57.736 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T17:58:57.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T17:58:57.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T17:58:57.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T17:58:57.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T17:58:57.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T17:58:57.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T17:58:58.680 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T17:58:58.681 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:06:57 ago on Fri 14 Jul 2023 05:52:01 PM UTC. 2023-07-14T17:58:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T17:58:58.911 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:06:52 ago on Fri 14 Jul 2023 05:52:06 PM UTC. 2023-07-14T17:59:05.877 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-07-14T17:59:06.085 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T17:59:06.085 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-07-14T17:59:06.086 INFO:teuthology.orchestra.run.smithi150.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:06.086 INFO:teuthology.orchestra.run.smithi150.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:59:06.086 INFO:teuthology.orchestra.run.smithi150.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:06.086 INFO:teuthology.orchestra.run.smithi150.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T17:59:06.086 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T17:59:06.086 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T17:59:06.086 INFO:teuthology.orchestra.run.smithi150.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:06.087 INFO:teuthology.orchestra.run.smithi150.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:06.087 INFO:teuthology.orchestra.run.smithi150.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:59:06.087 INFO:teuthology.orchestra.run.smithi150.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T17:59:06.087 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T17:59:06.087 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-07-14T17:59:06.236 DEBUG:teuthology.parallel:result is None 2023-07-14T17:59:09.044 INFO:teuthology.orchestra.run.smithi115.stdout:Installed products updated. 2023-07-14T17:59:09.252 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T17:59:09.252 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-07-14T17:59:09.253 INFO:teuthology.orchestra.run.smithi115.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:09.253 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:59:09.253 INFO:teuthology.orchestra.run.smithi115.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:09.254 INFO:teuthology.orchestra.run.smithi115.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T17:59:09.254 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T17:59:09.254 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T17:59:09.254 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:09.255 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:09.255 INFO:teuthology.orchestra.run.smithi115.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:59:09.256 INFO:teuthology.orchestra.run.smithi115.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T17:59:09.256 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T17:59:09.256 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-07-14T17:59:09.434 DEBUG:teuthology.parallel:result is None 2023-07-14T17:59:10.186 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-07-14T17:59:10.403 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T17:59:10.403 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-07-14T17:59:10.403 INFO:teuthology.orchestra.run.smithi187.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:59:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T17:59:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T17:59:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T17:59:10.404 INFO:teuthology.orchestra.run.smithi187.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:10.405 INFO:teuthology.orchestra.run.smithi187.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:59:10.405 INFO:teuthology.orchestra.run.smithi187.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:59:10.405 INFO:teuthology.orchestra.run.smithi187.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T17:59:10.405 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T17:59:10.405 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-07-14T17:59:10.577 DEBUG:teuthology.parallel:result is None 2023-07-14T17:59:10.577 INFO:teuthology.run_tasks:Running task install... 2023-07-14T17:59:10.589 DEBUG:teuthology.task.install:project ceph 2023-07-14T17:59:10.590 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'}} 2023-07-14T17:59:10.590 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'} 2023-07-14T17:59:10.590 INFO:teuthology.task.install:Using flavor: default 2023-07-14T17:59:10.597 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-14T17:59:10.597 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-14T17:59:10.598 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T17:59:10.598 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:59:10.601 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T17:59:10.601 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:59:10.603 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T17:59:10.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:59:25.898 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T17:59:25.898 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T17:59:30.909 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T17:59:30.909 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T17:59:30.911 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T17:59:30.911 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T17:59:41.155 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T17:59:41.155 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T17:59:41.155 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T17:59:41.158 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T17:59:41.158 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T17:59:41.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T17:59:41.192 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T17:59:41.193 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T17:59:41.194 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T17:59:41.194 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T17:59:41.257 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T17:59:41.257 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T17:59:41.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T17:59:41.265 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T17:59:41.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T17:59:41.293 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T17:59:41.293 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T17:59:41.346 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T17:59:41.347 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T17:59:41.363 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T17:59:41.413 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2023-07-14T17:59:41.415 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2023-07-14T17:59:41.416 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2023-07-14T17:59:41.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2023-07-14T17:59:41.443 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T17:59:41.510 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2023-07-14T17:59:41.512 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2023-07-14T17:59:41.787 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:41.791 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:41.884 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:42.730 INFO:teuthology.orchestra.run.smithi115.stdout:44 files removed 2023-07-14T17:59:42.764 INFO:teuthology.orchestra.run.smithi187.stdout:44 files removed 2023-07-14T17:59:42.783 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T17:59:42.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T17:59:42.855 INFO:teuthology.orchestra.run.smithi150.stdout:44 files removed 2023-07-14T17:59:42.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T17:59:43.115 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:43.150 INFO:teuthology.orchestra.run.smithi187.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:43.244 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:45.366 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 78 kB/s | 81 kB 00:01 2023-07-14T17:59:45.418 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 78 kB/s | 81 kB 00:01 2023-07-14T17:59:45.707 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 64 kB/s | 81 kB 00:01 2023-07-14T17:59:46.375 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 9.7 kB/s | 9.4 kB 00:00 2023-07-14T17:59:46.435 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 9.6 kB/s | 9.4 kB 00:00 2023-07-14T17:59:46.774 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 9.1 kB/s | 9.4 kB 00:01 2023-07-14T17:59:47.418 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 1.5 kB/s | 1.5 kB 00:01 2023-07-14T17:59:47.436 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 1.5 kB/s | 1.5 kB 00:00 2023-07-14T17:59:47.750 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 1.6 kB/s | 1.5 kB 00:00 2023-07-14T17:59:47.977 INFO:teuthology.orchestra.run.smithi187.stdout:Copr repo for python3-asyncssh owned by ceph 6.4 kB/s | 3.5 kB 00:00 2023-07-14T17:59:48.003 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 6.3 kB/s | 3.5 kB 00:00 2023-07-14T17:59:48.342 INFO:teuthology.orchestra.run.smithi150.stdout:Copr repo for python3-asyncssh owned by ceph 6.1 kB/s | 3.5 kB 00:00 2023-07-14T17:59:51.222 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 4.9 MB/s | 16 MB 00:03 2023-07-14T17:59:52.460 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 16 MB 00:04 2023-07-14T17:59:54.653 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 413 kB/s | 24 kB 00:00 2023-07-14T17:59:55.780 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-07-14T17:59:55.877 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 611 kB/s | 24 kB 00:00 2023-07-14T17:59:57.038 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 47 MB 00:01 2023-07-14T18:00:03.823 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 53 MB/s | 53 MB 00:01 2023-07-14T18:00:05.096 INFO:teuthology.orchestra.run.smithi187.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 53 MB 00:01 2023-07-14T18:00:14.900 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:00:14.901 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:00:14.916 INFO:teuthology.orchestra.run.smithi115.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T18:00:14.917 INFO:teuthology.orchestra.run.smithi115.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T18:00:14.918 INFO:teuthology.orchestra.run.smithi115.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T18:00:14.919 INFO:teuthology.orchestra.run.smithi115.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T18:00:14.921 INFO:teuthology.orchestra.run.smithi115.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T18:00:14.924 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T18:00:14.926 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T18:00:15.056 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-07-14T18:00:15.068 INFO:teuthology.orchestra.run.smithi115.stdout:=========================================================================================================================== 2023-07-14T18:00:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-07-14T18:00:15.069 INFO:teuthology.orchestra.run.smithi115.stdout:=========================================================================================================================== 2023-07-14T18:00:15.069 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-07-14T18:00:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T18:00:15.069 INFO:teuthology.orchestra.run.smithi115.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T18:00:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T18:00:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T18:00:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T18:00:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T18:00:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T18:00:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T18:00:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T18:00:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T18:00:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T18:00:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T18:00:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T18:00:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T18:00:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T18:00:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T18:00:15.072 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T18:00:15.072 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T18:00:15.072 INFO:teuthology.orchestra.run.smithi115.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T18:00:15.072 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T18:00:15.072 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T18:00:15.072 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T18:00:15.072 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T18:00:15.073 INFO:teuthology.orchestra.run.smithi115.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T18:00:15.073 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T18:00:15.073 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T18:00:15.073 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T18:00:15.073 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T18:00:15.073 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T18:00:15.073 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T18:00:15.074 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T18:00:15.074 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T18:00:15.074 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T18:00:15.074 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T18:00:15.074 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T18:00:15.074 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T18:00:15.075 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T18:00:15.075 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2023-07-14T18:00:15.075 INFO:teuthology.orchestra.run.smithi115.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T18:00:15.075 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T18:00:15.075 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T18:00:15.075 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-07-14T18:00:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T18:00:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T18:00:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T18:00:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T18:00:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T18:00:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T18:00:15.076 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T18:00:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T18:00:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T18:00:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T18:00:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T18:00:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T18:00:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T18:00:15.077 INFO:teuthology.orchestra.run.smithi115.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T18:00:15.078 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T18:00:15.078 INFO:teuthology.orchestra.run.smithi115.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T18:00:15.078 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T18:00:15.078 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T18:00:15.078 INFO:teuthology.orchestra.run.smithi115.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T18:00:15.078 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T18:00:15.078 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T18:00:15.079 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T18:00:15.079 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T18:00:15.079 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T18:00:15.079 INFO:teuthology.orchestra.run.smithi115.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T18:00:15.079 INFO:teuthology.orchestra.run.smithi115.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T18:00:15.080 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T18:00:15.080 INFO:teuthology.orchestra.run.smithi115.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T18:00:15.080 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T18:00:15.080 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T18:00:15.080 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T18:00:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T18:00:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T18:00:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T18:00:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T18:00:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T18:00:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T18:00:15.081 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T18:00:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T18:00:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T18:00:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T18:00:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T18:00:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T18:00:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T18:00:15.082 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T18:00:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T18:00:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T18:00:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T18:00:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T18:00:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T18:00:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T18:00:15.083 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T18:00:15.084 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T18:00:15.084 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:00:15.084 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T18:00:15.084 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T18:00:15.084 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T18:00:15.084 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T18:00:15.084 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T18:00:15.085 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T18:00:15.085 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T18:00:15.085 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T18:00:15.085 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T18:00:15.085 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T18:00:15.085 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T18:00:15.085 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T18:00:15.086 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T18:00:15.086 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T18:00:15.086 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T18:00:15.086 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T18:00:15.086 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T18:00:15.086 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T18:00:15.086 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:00:15.087 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T18:00:15.087 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T18:00:15.087 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T18:00:15.087 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T18:00:15.087 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T18:00:15.087 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T18:00:15.087 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T18:00:15.088 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T18:00:15.088 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T18:00:15.088 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T18:00:15.088 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T18:00:15.088 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T18:00:15.088 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T18:00:15.088 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T18:00:15.089 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T18:00:15.089 INFO:teuthology.orchestra.run.smithi115.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T18:00:15.089 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T18:00:15.089 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T18:00:15.089 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling module streams: 2023-07-14T18:00:15.089 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql 10 2023-07-14T18:00:15.090 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:00:15.090 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-07-14T18:00:15.090 INFO:teuthology.orchestra.run.smithi115.stdout:=========================================================================================================================== 2023-07-14T18:00:15.090 INFO:teuthology.orchestra.run.smithi115.stdout:Install 128 Packages 2023-07-14T18:00:15.090 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 3 Packages 2023-07-14T18:00:15.090 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:00:15.091 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 206 M 2023-07-14T18:00:15.091 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-07-14T18:00:15.764 INFO:teuthology.orchestra.run.smithi115.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 10 kB/s | 6.4 kB 00:00 2023-07-14T18:00:15.835 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 582 kB/s | 16 MB 00:27 2023-07-14T18:00:16.322 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:00:16.323 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:00:16.337 INFO:teuthology.orchestra.run.smithi187.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T18:00:16.338 INFO:teuthology.orchestra.run.smithi187.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T18:00:16.339 INFO:teuthology.orchestra.run.smithi187.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T18:00:16.341 INFO:teuthology.orchestra.run.smithi187.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T18:00:16.343 INFO:teuthology.orchestra.run.smithi187.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T18:00:16.346 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T18:00:16.347 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T18:00:16.480 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-07-14T18:00:16.493 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================================================== 2023-07-14T18:00:16.493 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-07-14T18:00:16.493 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================================================== 2023-07-14T18:00:16.493 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-07-14T18:00:16.493 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T18:00:16.494 INFO:teuthology.orchestra.run.smithi187.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T18:00:16.494 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T18:00:16.494 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T18:00:16.494 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T18:00:16.494 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T18:00:16.494 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T18:00:16.494 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T18:00:16.494 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T18:00:16.495 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T18:00:16.495 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T18:00:16.495 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T18:00:16.495 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T18:00:16.495 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T18:00:16.495 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T18:00:16.495 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T18:00:16.495 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T18:00:16.496 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T18:00:16.496 INFO:teuthology.orchestra.run.smithi187.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T18:00:16.496 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T18:00:16.496 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T18:00:16.496 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T18:00:16.496 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T18:00:16.497 INFO:teuthology.orchestra.run.smithi187.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T18:00:16.497 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T18:00:16.497 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T18:00:16.497 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T18:00:16.497 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T18:00:16.497 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T18:00:16.497 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T18:00:16.497 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T18:00:16.498 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T18:00:16.498 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T18:00:16.498 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T18:00:16.498 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T18:00:16.498 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T18:00:16.498 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T18:00:16.498 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2023-07-14T18:00:16.499 INFO:teuthology.orchestra.run.smithi187.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T18:00:16.499 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T18:00:16.499 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T18:00:16.499 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-07-14T18:00:16.499 INFO:teuthology.orchestra.run.smithi187.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T18:00:16.499 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T18:00:16.499 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T18:00:16.499 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T18:00:16.500 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T18:00:16.500 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T18:00:16.500 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T18:00:16.500 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T18:00:16.500 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T18:00:16.500 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T18:00:16.500 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T18:00:16.501 INFO:teuthology.orchestra.run.smithi187.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T18:00:16.501 INFO:teuthology.orchestra.run.smithi187.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T18:00:16.501 INFO:teuthology.orchestra.run.smithi187.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T18:00:16.501 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T18:00:16.501 INFO:teuthology.orchestra.run.smithi187.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T18:00:16.501 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T18:00:16.501 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T18:00:16.501 INFO:teuthology.orchestra.run.smithi187.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T18:00:16.502 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T18:00:16.502 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T18:00:16.502 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T18:00:16.502 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T18:00:16.502 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T18:00:16.502 INFO:teuthology.orchestra.run.smithi187.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T18:00:16.502 INFO:teuthology.orchestra.run.smithi187.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T18:00:16.503 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T18:00:16.503 INFO:teuthology.orchestra.run.smithi187.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T18:00:16.503 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T18:00:16.503 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T18:00:16.503 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T18:00:16.503 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T18:00:16.503 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T18:00:16.504 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T18:00:16.504 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T18:00:16.504 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T18:00:16.504 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T18:00:16.504 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T18:00:16.504 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T18:00:16.504 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T18:00:16.504 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T18:00:16.505 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T18:00:16.505 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T18:00:16.505 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T18:00:16.505 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T18:00:16.505 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T18:00:16.505 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T18:00:16.505 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T18:00:16.506 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T18:00:16.506 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T18:00:16.506 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T18:00:16.506 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T18:00:16.506 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T18:00:16.506 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:00:16.506 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T18:00:16.506 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T18:00:16.507 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T18:00:16.507 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T18:00:16.507 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T18:00:16.507 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T18:00:16.507 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T18:00:16.507 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T18:00:16.507 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T18:00:16.507 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T18:00:16.508 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T18:00:16.508 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T18:00:16.508 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T18:00:16.508 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T18:00:16.508 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T18:00:16.508 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T18:00:16.508 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T18:00:16.509 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T18:00:16.509 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:00:16.509 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T18:00:16.509 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T18:00:16.509 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T18:00:16.509 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T18:00:16.509 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T18:00:16.510 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T18:00:16.510 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T18:00:16.510 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T18:00:16.510 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T18:00:16.510 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T18:00:16.510 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T18:00:16.510 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T18:00:16.510 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T18:00:16.511 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T18:00:16.511 INFO:teuthology.orchestra.run.smithi187.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T18:00:16.511 INFO:teuthology.orchestra.run.smithi187.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T18:00:16.511 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T18:00:16.511 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T18:00:16.511 INFO:teuthology.orchestra.run.smithi187.stdout:Enabling module streams: 2023-07-14T18:00:16.511 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql 10 2023-07-14T18:00:16.512 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T18:00:16.512 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-07-14T18:00:16.512 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================================================== 2023-07-14T18:00:16.512 INFO:teuthology.orchestra.run.smithi187.stdout:Install 128 Packages 2023-07-14T18:00:16.512 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 3 Packages 2023-07-14T18:00:16.512 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T18:00:16.513 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 206 M 2023-07-14T18:00:16.513 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-07-14T18:00:16.665 INFO:teuthology.orchestra.run.smithi115.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 945 kB/s | 850 kB 00:00 2023-07-14T18:00:16.823 INFO:teuthology.orchestra.run.smithi115.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 1.0 MB/s | 167 kB 00:00 2023-07-14T18:00:17.065 INFO:teuthology.orchestra.run.smithi115.stdout:(4/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 2.8 MB/s | 5.4 MB 00:01 2023-07-14T18:00:17.340 INFO:teuthology.orchestra.run.smithi115.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 5.6 MB/s | 1.6 MB 00:00 2023-07-14T18:00:17.615 INFO:teuthology.orchestra.run.smithi115.stdout:(6/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 2.9 MB/s | 2.3 MB 00:00 2023-07-14T18:00:17.705 INFO:teuthology.orchestra.run.smithi187.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 13 kB/s | 6.4 kB 00:00 2023-07-14T18:00:17.899 INFO:teuthology.orchestra.run.smithi115.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 8.3 MB/s | 4.6 MB 00:00 2023-07-14T18:00:18.947 INFO:teuthology.orchestra.run.smithi187.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 685 kB/s | 850 kB 00:01 2023-07-14T18:00:18.999 INFO:teuthology.orchestra.run.smithi115.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 5.7 MB/s | 22 MB 00:03 2023-07-14T18:00:19.166 INFO:teuthology.orchestra.run.smithi115.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 146 kB/s | 24 kB 00:00 2023-07-14T18:00:19.206 INFO:teuthology.orchestra.run.smithi187.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 647 kB/s | 167 kB 00:00 2023-07-14T18:00:19.292 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 397 kB/s | 24 kB 00:00 2023-07-14T18:00:19.825 INFO:teuthology.orchestra.run.smithi115.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 7.0 MB/s | 14 MB 00:01 2023-07-14T18:00:19.975 INFO:teuthology.orchestra.run.smithi115.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 1.7 MB/s | 260 kB 00:00 2023-07-14T18:00:20.159 INFO:teuthology.orchestra.run.smithi115.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 169 kB/s | 31 kB 00:00 2023-07-14T18:00:20.427 INFO:teuthology.orchestra.run.smithi150.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-07-14T18:00:20.647 INFO:teuthology.orchestra.run.smithi187.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 1.6 MB/s | 2.3 MB 00:01 2023-07-14T18:00:21.223 INFO:teuthology.orchestra.run.smithi187.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 2.7 MB/s | 1.6 MB 00:00 2023-07-14T18:00:21.234 INFO:teuthology.orchestra.run.smithi115.stdout:(13/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 4.9 MB/s | 18 MB 00:03 2023-07-14T18:00:21.384 INFO:teuthology.orchestra.run.smithi115.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 905 kB/s | 135 kB 00:00 2023-07-14T18:00:21.418 INFO:teuthology.orchestra.run.smithi115.stdout:(15/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 614 kB/s | 772 kB 00:01 2023-07-14T18:00:21.518 INFO:teuthology.orchestra.run.smithi115.stdout:(16/131): librados-devel-18.0.0-4940.g81c9387d. 1.0 MB/s | 139 kB 00:00 2023-07-14T18:00:21.560 INFO:teuthology.orchestra.run.smithi115.stdout:(17/131): libradosstriper1-18.0.0-4940.g81c9387 3.2 MB/s | 473 kB 00:00 2023-07-14T18:00:21.736 INFO:teuthology.orchestra.run.smithi115.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 255 kB/s | 45 kB 00:00 2023-07-14T18:00:21.894 INFO:teuthology.orchestra.run.smithi115.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 774 kB/s | 122 kB 00:00 2023-07-14T18:00:22.053 INFO:teuthology.orchestra.run.smithi115.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 1.2 MB/s | 197 kB 00:00 2023-07-14T18:00:22.219 INFO:teuthology.orchestra.run.smithi115.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 2.2 MB/s | 371 kB 00:00 2023-07-14T18:00:22.378 INFO:teuthology.orchestra.run.smithi115.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 2.2 MB/s | 364 kB 00:00 2023-07-14T18:00:22.565 INFO:teuthology.orchestra.run.smithi187.stdout:(6/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 3.4 MB/s | 4.6 MB 00:01 2023-07-14T18:00:22.570 INFO:teuthology.orchestra.run.smithi115.stdout:(23/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 6.4 MB/s | 6.8 MB 00:01 2023-07-14T18:00:22.595 INFO:teuthology.orchestra.run.smithi115.stdout:(24/131): python3-rgw-18.0.0-4940.g81c9387d.el8 492 kB/s | 107 kB 00:00 2023-07-14T18:00:22.746 INFO:teuthology.orchestra.run.smithi115.stdout:(25/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 535 kB/s | 94 kB 00:00 2023-07-14T18:00:22.905 INFO:teuthology.orchestra.run.smithi115.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.1 MB/s | 179 kB 00:00 2023-07-14T18:00:23.063 INFO:teuthology.orchestra.run.smithi115.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 146 kB/s | 23 kB 00:00 2023-07-14T18:00:23.090 INFO:teuthology.orchestra.run.smithi187.stdout:(7/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 937 kB/s | 5.4 MB 00:05 2023-07-14T18:00:23.530 INFO:teuthology.orchestra.run.smithi115.stdout:(28/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 11 MB/s | 48 MB 00:04 2023-07-14T18:00:23.555 INFO:teuthology.orchestra.run.smithi115.stdout:(29/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 263 kB/s | 129 kB 00:00 2023-07-14T18:00:23.630 INFO:teuthology.orchestra.run.smithi115.stdout:(30/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 3.4 MB/s | 3.5 MB 00:01 2023-07-14T18:00:23.773 INFO:teuthology.orchestra.run.smithi115.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 1.7 MB/s | 241 kB 00:00 2023-07-14T18:00:23.815 INFO:teuthology.orchestra.run.smithi115.stdout:(32/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 6.0 MB/s | 1.7 MB 00:00 2023-07-14T18:00:23.898 INFO:teuthology.orchestra.run.smithi115.stdout:(33/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 379 kB/s | 47 kB 00:00 2023-07-14T18:00:23.957 INFO:teuthology.orchestra.run.smithi115.stdout:(34/131): ceph-prometheus-alerts-18.0.0-4940.g8 102 kB/s | 15 kB 00:00 2023-07-14T18:00:24.024 INFO:teuthology.orchestra.run.smithi115.stdout:(35/131): ceph-volume-18.0.0-4940.g81c9387d.el8 2.0 MB/s | 258 kB 00:00 2023-07-14T18:00:24.091 INFO:teuthology.orchestra.run.smithi115.stdout:(36/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 1.2 MB/s | 165 kB 00:00 2023-07-14T18:00:24.174 INFO:teuthology.orchestra.run.smithi115.stdout:(37/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 120 kB/s | 18 kB 00:00 2023-07-14T18:00:24.516 INFO:teuthology.orchestra.run.smithi115.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 7.7 MB/s | 7.4 MB 00:00 2023-07-14T18:00:24.642 INFO:teuthology.orchestra.run.smithi115.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 126 kB/s | 59 kB 00:00 2023-07-14T18:00:24.773 INFO:teuthology.orchestra.run.smithi187.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 2.9 MB/s | 22 MB 00:07 2023-07-14T18:00:24.883 INFO:teuthology.orchestra.run.smithi115.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 386 kB/s | 306 kB 00:00 2023-07-14T18:00:24.974 INFO:teuthology.orchestra.run.smithi187.stdout:(9/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x86 7.4 MB/s | 18 MB 00:02 2023-07-14T18:00:24.990 INFO:teuthology.orchestra.run.smithi187.stdout:(10/131): ceph-selinux-18.0.0-4940.g81c9387d.el 112 kB/s | 24 kB 00:00 2023-07-14T18:00:25.017 INFO:teuthology.orchestra.run.smithi115.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 327 kB/s | 44 kB 00:00 2023-07-14T18:00:25.042 INFO:teuthology.orchestra.run.smithi115.stdout:(42/131): libunwind-1.3.1-3.el8.x86_64.rpm 143 kB/s | 75 kB 00:00 2023-07-14T18:00:25.109 INFO:teuthology.orchestra.run.smithi115.stdout:(43/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.0 MB/s | 476 kB 00:00 2023-07-14T18:00:25.149 INFO:teuthology.orchestra.run.smithi187.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 1.6 MB/s | 260 kB 00:00 2023-07-14T18:00:25.177 INFO:teuthology.orchestra.run.smithi115.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.1 MB/s | 185 kB 00:00 2023-07-14T18:00:25.202 INFO:teuthology.orchestra.run.smithi115.stdout:(45/131): python3-cachetools-3.1.1-4.el8.noarch 207 kB/s | 33 kB 00:00 2023-07-14T18:00:25.227 INFO:teuthology.orchestra.run.smithi115.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 139 kB/s | 16 kB 00:00 2023-07-14T18:00:25.266 INFO:teuthology.orchestra.run.smithi187.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 266 kB/s | 31 kB 00:00 2023-07-14T18:00:25.344 INFO:teuthology.orchestra.run.smithi115.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.0 MB/s | 173 kB 00:00 2023-07-14T18:00:25.369 INFO:teuthology.orchestra.run.smithi115.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 572 kB/s | 81 kB 00:00 2023-07-14T18:00:25.453 INFO:teuthology.orchestra.run.smithi115.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-07-14T18:00:25.478 INFO:teuthology.orchestra.run.smithi115.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 162 kB/s | 18 kB 00:00 2023-07-14T18:00:25.491 INFO:teuthology.orchestra.run.smithi187.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 3.4 MB/s | 772 kB 00:00 2023-07-14T18:00:25.603 INFO:teuthology.orchestra.run.smithi115.stdout:(51/131): python3-cherrypy-18.4.0-1.el8.noarch. 958 kB/s | 384 kB 00:00 2023-07-14T18:00:25.629 INFO:teuthology.orchestra.run.smithi115.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 324 kB/s | 49 kB 00:00 2023-07-14T18:00:25.659 INFO:teuthology.orchestra.run.smithi187.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 813 kB/s | 135 kB 00:00 2023-07-14T18:00:25.737 INFO:teuthology.orchestra.run.smithi115.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 440 kB/s | 59 kB 00:00 2023-07-14T18:00:25.763 INFO:teuthology.orchestra.run.smithi115.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 451 kB/s | 60 kB 00:00 2023-07-14T18:00:25.792 INFO:teuthology.orchestra.run.smithi187.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 1.0 MB/s | 139 kB 00:00 2023-07-14T18:00:25.906 INFO:teuthology.orchestra.run.smithi115.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 2.6 MB/s | 1.2 MB 00:00 2023-07-14T18:00:25.938 INFO:teuthology.orchestra.run.smithi115.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 89 kB/s | 16 kB 00:00 2023-07-14T18:00:25.963 INFO:teuthology.orchestra.run.smithi115.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2023-07-14T18:00:26.022 INFO:teuthology.orchestra.run.smithi115.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 292 kB/s | 34 kB 00:00 2023-07-14T18:00:26.034 INFO:teuthology.orchestra.run.smithi187.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 1.9 MB/s | 473 kB 00:00 2023-07-14T18:00:26.080 INFO:teuthology.orchestra.run.smithi115.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.3 MB/s | 196 kB 00:00 2023-07-14T18:00:26.105 INFO:teuthology.orchestra.run.smithi115.stdout:(60/131): python3-rsa-4.8-1.el8.noarch.rpm 433 kB/s | 61 kB 00:00 2023-07-14T18:00:26.148 INFO:teuthology.orchestra.run.smithi115.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 149 kB/s | 19 kB 00:00 2023-07-14T18:00:26.231 INFO:teuthology.orchestra.run.smithi115.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 191 kB/s | 24 kB 00:00 2023-07-14T18:00:26.256 INFO:teuthology.orchestra.run.smithi115.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 194 kB/s | 29 kB 00:00 2023-07-14T18:00:26.282 INFO:teuthology.orchestra.run.smithi115.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 206 kB/s | 27 kB 00:00 2023-07-14T18:00:26.382 INFO:teuthology.orchestra.run.smithi115.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.7 MB/s | 255 kB 00:00 2023-07-14T18:00:26.407 INFO:teuthology.orchestra.run.smithi115.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 488 kB/s | 61 kB 00:00 2023-07-14T18:00:26.433 INFO:teuthology.orchestra.run.smithi115.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.4 MB/s | 251 kB 00:00 2023-07-14T18:00:26.541 INFO:teuthology.orchestra.run.smithi115.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 537 kB/s | 85 kB 00:00 2023-07-14T18:00:26.567 INFO:teuthology.orchestra.run.smithi115.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 144 kB/s | 23 kB 00:00 2023-07-14T18:00:26.591 INFO:teuthology.orchestra.run.smithi115.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-07-14T18:00:26.967 INFO:teuthology.orchestra.run.smithi115.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 3.2 MB/s | 1.7 MB 00:00 2023-07-14T18:00:26.992 INFO:teuthology.orchestra.run.smithi115.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 275 kB/s | 110 kB 00:00 2023-07-14T18:00:27.210 INFO:teuthology.orchestra.run.smithi115.stdout:(73/131): python3-scipy-1.0.0-21.module+el8.5.0 21 MB/s | 14 MB 00:00 2023-07-14T18:00:27.235 INFO:teuthology.orchestra.run.smithi115.stdout:(74/131): python3-pytz-2017.2-9.el8.noarch.rpm 203 kB/s | 54 kB 00:00 2023-07-14T18:00:27.260 INFO:teuthology.orchestra.run.smithi115.stdout:(75/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 989 kB/s | 259 kB 00:00 2023-07-14T18:00:27.360 INFO:teuthology.orchestra.run.smithi115.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 6.7 MB/s | 688 kB 00:00 2023-07-14T18:00:27.385 INFO:teuthology.orchestra.run.smithi115.stdout:(77/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2023-07-14T18:00:27.402 INFO:teuthology.orchestra.run.smithi115.stdout:(78/131): python3-markupsafe-0.23-19.el8.x86_64 232 kB/s | 39 kB 00:00 2023-07-14T18:00:27.494 INFO:teuthology.orchestra.run.smithi115.stdout:(79/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.9 MB/s | 538 kB 00:00 2023-07-14T18:00:27.578 INFO:teuthology.orchestra.run.smithi115.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-07-14T18:00:27.628 INFO:teuthology.orchestra.run.smithi115.stdout:(81/131): python3-mako-1.0.6-13.el8.noarch.rpm 649 kB/s | 157 kB 00:00 2023-07-14T18:00:27.670 INFO:teuthology.orchestra.run.smithi115.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 4.0 MB/s | 713 kB 00:00 2023-07-14T18:00:27.721 INFO:teuthology.orchestra.run.smithi115.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 286 kB/s | 27 kB 00:00 2023-07-14T18:00:27.796 INFO:teuthology.orchestra.run.smithi115.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2023-07-14T18:00:27.821 INFO:teuthology.orchestra.run.smithi115.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 2.1 MB/s | 320 kB 00:00 2023-07-14T18:00:27.859 INFO:teuthology.orchestra.run.smithi187.stdout:(17/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 17 MB/s | 48 MB 00:02 2023-07-14T18:00:27.888 INFO:teuthology.orchestra.run.smithi115.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 263 kB/s | 44 kB 00:00 2023-07-14T18:00:27.951 INFO:teuthology.orchestra.run.smithi187.stdout:(18/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 3.5 MB/s | 6.8 MB 00:01 2023-07-14T18:00:27.985 INFO:teuthology.orchestra.run.smithi187.stdout:(19/131): python3-ceph-argparse-18.0.0-4940.g81 358 kB/s | 45 kB 00:00 2023-07-14T18:00:27.988 INFO:teuthology.orchestra.run.smithi115.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 19 MB/s | 3.7 MB 00:00 2023-07-14T18:00:28.014 INFO:teuthology.orchestra.run.smithi115.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.8 MB/s | 225 kB 00:00 2023-07-14T18:00:28.047 INFO:teuthology.orchestra.run.smithi115.stdout:(89/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.0 MB/s | 457 kB 00:00 2023-07-14T18:00:28.076 INFO:teuthology.orchestra.run.smithi187.stdout:(20/131): python3-ceph-common-18.0.0-4940.g81c9 979 kB/s | 122 kB 00:00 2023-07-14T18:00:28.097 INFO:teuthology.orchestra.run.smithi115.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 589 kB/s | 64 kB 00:00 2023-07-14T18:00:28.110 INFO:teuthology.orchestra.run.smithi187.stdout:(21/131): python3-cephfs-18.0.0-4940.g81c9387d. 1.5 MB/s | 197 kB 00:00 2023-07-14T18:00:28.148 INFO:teuthology.orchestra.run.smithi115.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 945 kB/s | 126 kB 00:00 2023-07-14T18:00:28.215 INFO:teuthology.orchestra.run.smithi115.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 119 kB/s | 20 kB 00:00 2023-07-14T18:00:28.219 INFO:teuthology.orchestra.run.smithi187.stdout:(22/131): python3-rados-18.0.0-4940.g81c9387d.e 2.6 MB/s | 371 kB 00:00 2023-07-14T18:00:28.240 INFO:teuthology.orchestra.run.smithi115.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 728 kB/s | 103 kB 00:00 2023-07-14T18:00:28.244 INFO:teuthology.orchestra.run.smithi187.stdout:(23/131): python3-rbd-18.0.0-4940.g81c9387d.el8 2.7 MB/s | 364 kB 00:00 2023-07-14T18:00:28.274 INFO:teuthology.orchestra.run.smithi115.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 5.5 MB/s | 709 kB 00:00 2023-07-14T18:00:28.315 INFO:teuthology.orchestra.run.smithi115.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.6 MB/s | 198 kB 00:00 2023-07-14T18:00:28.344 INFO:teuthology.orchestra.run.smithi187.stdout:(24/131): python3-rgw-18.0.0-4940.g81c9387d.el8 851 kB/s | 107 kB 00:00 2023-07-14T18:00:28.378 INFO:teuthology.orchestra.run.smithi187.stdout:(25/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 700 kB/s | 94 kB 00:00 2023-07-14T18:00:28.449 INFO:teuthology.orchestra.run.smithi115.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 21 MB/s | 4.9 MB 00:00 2023-07-14T18:00:28.474 INFO:teuthology.orchestra.run.smithi115.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-07-14T18:00:28.520 INFO:teuthology.orchestra.run.smithi187.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.2 MB/s | 179 kB 00:00 2023-07-14T18:00:28.575 INFO:teuthology.orchestra.run.smithi115.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-07-14T18:00:28.625 INFO:teuthology.orchestra.run.smithi115.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.0 MB/s | 354 kB 00:00 2023-07-14T18:00:28.662 INFO:teuthology.orchestra.run.smithi187.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 163 kB/s | 23 kB 00:00 2023-07-14T18:00:28.667 INFO:teuthology.orchestra.run.smithi115.stdout:(100/131): postgresql-contrib-10.21-2.module+el 8.6 MB/s | 808 kB 00:00 2023-07-14T18:00:28.703 INFO:teuthology.orchestra.run.smithi150.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 53 MB 00:01 2023-07-14T18:00:28.804 INFO:teuthology.orchestra.run.smithi187.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 911 kB/s | 129 kB 00:00 2023-07-14T18:00:28.817 INFO:teuthology.orchestra.run.smithi115.stdout:(101/131): postgresql-server-10.21-2.module+el8 15 MB/s | 5.1 MB 00:00 2023-07-14T18:00:28.842 INFO:teuthology.orchestra.run.smithi115.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 267 kB/s | 47 kB 00:00 2023-07-14T18:00:28.893 INFO:teuthology.orchestra.run.smithi115.stdout:(103/131): postgresql-10.21-2.module+el8.6.0+15 5.6 MB/s | 1.5 MB 00:00 2023-07-14T18:00:28.935 INFO:teuthology.orchestra.run.smithi115.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 461 kB/s | 43 kB 00:00 2023-07-14T18:00:28.980 INFO:teuthology.orchestra.run.smithi115.stdout:(105/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.2 MB/s | 200 kB 00:00 2023-07-14T18:00:29.002 INFO:teuthology.orchestra.run.smithi115.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2023-07-14T18:00:29.061 INFO:teuthology.orchestra.run.smithi115.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 382 kB/s | 48 kB 00:00 2023-07-14T18:00:29.136 INFO:teuthology.orchestra.run.smithi115.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 548 kB/s | 87 kB 00:00 2023-07-14T18:00:29.161 INFO:teuthology.orchestra.run.smithi115.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 268 kB/s | 43 kB 00:00 2023-07-14T18:00:29.196 INFO:teuthology.orchestra.run.smithi187.stdout:(29/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 4.4 MB/s | 1.7 MB 00:00 2023-07-14T18:00:29.228 INFO:teuthology.orchestra.run.smithi115.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 653 kB/s | 109 kB 00:00 2023-07-14T18:00:29.262 INFO:teuthology.orchestra.run.smithi115.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2023-07-14T18:00:29.305 INFO:teuthology.orchestra.run.smithi187.stdout:(30/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 3.7 MB/s | 3.5 MB 00:00 2023-07-14T18:00:29.312 INFO:teuthology.orchestra.run.smithi115.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.6 MB/s | 560 kB 00:00 2023-07-14T18:00:29.354 INFO:teuthology.orchestra.run.smithi115.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 149 kB/s | 19 kB 00:00 2023-07-14T18:00:29.438 INFO:teuthology.orchestra.run.smithi187.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 1.8 MB/s | 241 kB 00:00 2023-07-14T18:00:29.446 INFO:teuthology.orchestra.run.smithi115.stdout:(114/131): libverto-devel-0.3.0-5.el8.x86_64.rp 195 kB/s | 18 kB 00:00 2023-07-14T18:00:29.471 INFO:teuthology.orchestra.run.smithi115.stdout:(115/131): python3-oauthlib-2.1.0-1.el8.noarch. 740 kB/s | 155 kB 00:00 2023-07-14T18:00:29.505 INFO:teuthology.orchestra.run.smithi115.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 2.8 MB/s | 559 kB 00:00 2023-07-14T18:00:29.564 INFO:teuthology.orchestra.run.smithi115.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.0 MB/s | 605 kB 00:00 2023-07-14T18:00:29.572 INFO:teuthology.orchestra.run.smithi187.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 356 kB/s | 47 kB 00:00 2023-07-14T18:00:29.597 INFO:teuthology.orchestra.run.smithi115.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.9 MB/s | 238 kB 00:00 2023-07-14T18:00:29.681 INFO:teuthology.orchestra.run.smithi115.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 618 kB/s | 72 kB 00:00 2023-07-14T18:00:29.697 INFO:teuthology.orchestra.run.smithi187.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 116 kB/s | 15 kB 00:00 2023-07-14T18:00:29.748 INFO:teuthology.orchestra.run.smithi115.stdout:(120/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 9.6 MB/s | 2.3 MB 00:00 2023-07-14T18:00:29.773 INFO:teuthology.orchestra.run.smithi115.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 471 kB/s | 83 kB 00:00 2023-07-14T18:00:29.798 INFO:teuthology.orchestra.run.smithi115.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 511 kB/s | 60 kB 00:00 2023-07-14T18:00:29.865 INFO:teuthology.orchestra.run.smithi115.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 627 kB/s | 58 kB 00:00 2023-07-14T18:00:29.889 INFO:teuthology.orchestra.run.smithi187.stdout:(34/131): ceph-mgr-diskprediction-local-18.0.0- 11 MB/s | 7.4 MB 00:00 2023-07-14T18:00:29.914 INFO:teuthology.orchestra.run.smithi187.stdout:(35/131): ceph-volume-18.0.0-4940.g81c9387d.el8 1.2 MB/s | 258 kB 00:00 2023-07-14T18:00:29.941 INFO:teuthology.orchestra.run.smithi115.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 272 kB/s | 39 kB 00:00 2023-07-14T18:00:29.966 INFO:teuthology.orchestra.run.smithi115.stdout:(125/131): libuuid-devel-2.32.1-35.el8.x86_64.r 450 kB/s | 98 kB 00:00 2023-07-14T18:00:29.991 INFO:teuthology.orchestra.run.smithi115.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 3.3 MB/s | 171 kB 00:00 2023-07-14T18:00:30.006 INFO:teuthology.orchestra.run.smithi187.stdout:(36/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 1.4 MB/s | 165 kB 00:00 2023-07-14T18:00:30.025 INFO:teuthology.orchestra.run.smithi115.stdout:(127/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 3.9 MB/s | 644 kB 00:00 2023-07-14T18:00:30.031 INFO:teuthology.orchestra.run.smithi187.stdout:(37/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 154 kB/s | 18 kB 00:00 2023-07-14T18:00:30.075 INFO:teuthology.orchestra.run.smithi115.stdout:(128/131): NetworkManager-initscripts-updown-1. 1.2 MB/s | 138 kB 00:00 2023-07-14T18:00:30.225 INFO:teuthology.orchestra.run.smithi115.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.2 MB/s | 798 kB 00:00 2023-07-14T18:00:30.574 INFO:teuthology.orchestra.run.smithi187.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 108 kB/s | 59 kB 00:00 2023-07-14T18:00:30.774 INFO:teuthology.orchestra.run.smithi187.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 399 kB/s | 306 kB 00:00 2023-07-14T18:00:30.825 INFO:teuthology.orchestra.run.smithi187.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 300 kB/s | 75 kB 00:00 2023-07-14T18:00:30.933 INFO:teuthology.orchestra.run.smithi187.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 402 kB/s | 44 kB 00:00 2023-07-14T18:00:30.984 INFO:teuthology.orchestra.run.smithi115.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 3.8 MB/s | 3.6 MB 00:00 2023-07-14T18:00:31.092 INFO:teuthology.orchestra.run.smithi187.stdout:(42/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 1.7 MB/s | 14 MB 00:08 2023-07-14T18:00:31.117 INFO:teuthology.orchestra.run.smithi187.stdout:(43/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.4 MB/s | 476 kB 00:00 2023-07-14T18:00:31.234 INFO:teuthology.orchestra.run.smithi187.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 616 kB/s | 185 kB 00:00 2023-07-14T18:00:31.267 INFO:teuthology.orchestra.run.smithi187.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 108 kB/s | 16 kB 00:00 2023-07-14T18:00:31.334 INFO:teuthology.orchestra.run.smithi115.stdout:(131/131): librados2-18.0.0-4940.g81c9387d.el8. 2.7 MB/s | 3.7 MB 00:01 2023-07-14T18:00:31.337 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:00:31.338 INFO:teuthology.orchestra.run.smithi115.stdout:Total 13 MB/s | 206 MB 00:16 2023-07-14T18:00:31.367 INFO:teuthology.orchestra.run.smithi187.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2023-07-14T18:00:31.451 INFO:teuthology.orchestra.run.smithi187.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.0 MB/s | 384 kB 00:00 2023-07-14T18:00:31.510 INFO:teuthology.orchestra.run.smithi187.stdout:(48/131): python3-cachetools-3.1.1-4.el8.noarch 79 kB/s | 33 kB 00:00 2023-07-14T18:00:31.535 INFO:teuthology.orchestra.run.smithi187.stdout:(49/131): python3-google-auth-1.1.1-10.el8.noar 486 kB/s | 81 kB 00:00 2023-07-14T18:00:31.602 INFO:teuthology.orchestra.run.smithi187.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 74 kB/s | 11 kB 00:00 2023-07-14T18:00:31.652 INFO:teuthology.orchestra.run.smithi187.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 124 kB/s | 18 kB 00:00 2023-07-14T18:00:31.752 INFO:teuthology.orchestra.run.smithi187.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 324 kB/s | 49 kB 00:00 2023-07-14T18:00:31.903 INFO:teuthology.orchestra.run.smithi187.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 402 kB/s | 60 kB 00:00 2023-07-14T18:00:31.928 INFO:teuthology.orchestra.run.smithi187.stdout:(54/131): python3-more-itertools-7.2.0-3.el8.no 213 kB/s | 59 kB 00:00 2023-07-14T18:00:32.070 INFO:teuthology.orchestra.run.smithi187.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-07-14T18:00:32.095 INFO:teuthology.orchestra.run.smithi187.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 93 kB/s | 16 kB 00:00 2023-07-14T18:00:32.129 INFO:teuthology.orchestra.run.smithi187.stdout:(57/131): python3-kubernetes-11.0.0-6.el8.noarc 2.0 MB/s | 1.2 MB 00:00 2023-07-14T18:00:32.175 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-07-14T18:00:32.238 INFO:teuthology.orchestra.run.smithi187.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 201 kB/s | 34 kB 00:00 2023-07-14T18:00:32.288 INFO:teuthology.orchestra.run.smithi187.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 388 kB/s | 61 kB 00:00 2023-07-14T18:00:32.295 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-07-14T18:00:32.295 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-07-14T18:00:32.465 INFO:teuthology.orchestra.run.smithi187.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 533 kB/s | 196 kB 00:00 2023-07-14T18:00:32.688 INFO:teuthology.orchestra.run.smithi187.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 42 kB/s | 19 kB 00:00 2023-07-14T18:00:32.831 INFO:teuthology.orchestra.run.smithi187.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 44 kB/s | 24 kB 00:00 2023-07-14T18:00:32.855 INFO:teuthology.orchestra.run.smithi187.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 75 kB/s | 29 kB 00:00 2023-07-14T18:00:32.973 INFO:teuthology.orchestra.run.smithi187.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 194 kB/s | 27 kB 00:00 2023-07-14T18:00:33.089 INFO:teuthology.orchestra.run.smithi187.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 988 kB/s | 255 kB 00:00 2023-07-14T18:00:33.115 INFO:teuthology.orchestra.run.smithi187.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 432 kB/s | 61 kB 00:00 2023-07-14T18:00:33.140 INFO:teuthology.orchestra.run.smithi187.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 884 kB/s | 251 kB 00:00 2023-07-14T18:00:33.240 INFO:teuthology.orchestra.run.smithi187.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 567 kB/s | 85 kB 00:00 2023-07-14T18:00:33.265 INFO:teuthology.orchestra.run.smithi187.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 152 kB/s | 23 kB 00:00 2023-07-14T18:00:33.291 INFO:teuthology.orchestra.run.smithi187.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-07-14T18:00:33.406 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-07-14T18:00:33.410 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-07-14T18:00:33.466 INFO:teuthology.orchestra.run.smithi187.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 651 kB/s | 110 kB 00:00 2023-07-14T18:00:33.650 INFO:teuthology.orchestra.run.smithi187.stdout:(72/131): thrift-0.13.0-2.el8.x86_64.rpm 3.4 MB/s | 1.7 MB 00:00 2023-07-14T18:00:33.667 INFO:teuthology.orchestra.run.smithi187.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 271 kB/s | 54 kB 00:00 2023-07-14T18:00:33.834 INFO:teuthology.orchestra.run.smithi187.stdout:(74/131): python3-scipy-1.0.0-21.module+el8.5.0 24 MB/s | 14 MB 00:00 2023-07-14T18:00:33.859 INFO:teuthology.orchestra.run.smithi187.stdout:(75/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2023-07-14T18:00:33.884 INFO:teuthology.orchestra.run.smithi187.stdout:(76/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2023-07-14T18:00:34.001 INFO:teuthology.orchestra.run.smithi187.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 4.5 MB/s | 538 kB 00:00 2023-07-14T18:00:34.076 INFO:teuthology.orchestra.run.smithi187.stdout:(78/131): python3-markupsafe-0.23-19.el8.x86_64 160 kB/s | 39 kB 00:00 2023-07-14T18:00:34.151 INFO:teuthology.orchestra.run.smithi187.stdout:(79/131): bison-3.0.4-10.el8.x86_64.rpm 2.3 MB/s | 688 kB 00:00 2023-07-14T18:00:34.177 INFO:teuthology.orchestra.run.smithi187.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 893 kB/s | 157 kB 00:00 2023-07-14T18:00:34.210 INFO:teuthology.orchestra.run.smithi187.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-07-14T18:00:34.244 INFO:teuthology.orchestra.run.smithi187.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 7.6 MB/s | 713 kB 00:00 2023-07-14T18:00:34.361 INFO:teuthology.orchestra.run.smithi187.stdout:(83/131): python3-babel-2.5.1-7.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2023-07-14T18:00:34.386 INFO:teuthology.orchestra.run.smithi187.stdout:(84/131): python3-jsonpatch-1.21-2.el8.noarch.r 151 kB/s | 27 kB 00:00 2023-07-14T18:00:34.420 INFO:teuthology.orchestra.run.smithi187.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 1.8 MB/s | 320 kB 00:00 2023-07-14T18:00:34.562 INFO:teuthology.orchestra.run.smithi187.stdout:(86/131): python3-numpy-1.14.3-10.el8.x86_64.rp 21 MB/s | 3.7 MB 00:00 2023-07-14T18:00:34.587 INFO:teuthology.orchestra.run.smithi187.stdout:(87/131): python3-prettytable-0.7.2-14.el8.noar 194 kB/s | 44 kB 00:00 2023-07-14T18:00:34.629 INFO:teuthology.orchestra.run.smithi187.stdout:(88/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.1 MB/s | 457 kB 00:00 2023-07-14T18:00:34.779 INFO:teuthology.orchestra.run.smithi187.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.0 MB/s | 225 kB 00:00 2023-07-14T18:00:34.813 INFO:teuthology.orchestra.run.smithi187.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 284 kB/s | 64 kB 00:00 2023-07-14T18:00:34.863 INFO:teuthology.orchestra.run.smithi187.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 239 kB/s | 20 kB 00:00 2023-07-14T18:00:34.889 INFO:teuthology.orchestra.run.smithi187.stdout:(92/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 488 kB/s | 126 kB 00:00 2023-07-14T18:00:34.989 INFO:teuthology.orchestra.run.smithi187.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 590 kB/s | 103 kB 00:00 2023-07-14T18:00:35.064 INFO:teuthology.orchestra.run.smithi187.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2023-07-14T18:00:35.098 INFO:teuthology.orchestra.run.smithi187.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.8 MB/s | 198 kB 00:00 2023-07-14T18:00:35.190 INFO:teuthology.orchestra.run.smithi187.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 16 MB/s | 4.9 MB 00:00 2023-07-14T18:00:35.215 INFO:teuthology.orchestra.run.smithi187.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.7 MB/s | 202 kB 00:00 2023-07-14T18:00:35.332 INFO:teuthology.orchestra.run.smithi187.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-07-14T18:00:35.365 INFO:teuthology.orchestra.run.smithi187.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.0 MB/s | 354 kB 00:00 2023-07-14T18:00:35.474 INFO:teuthology.orchestra.run.smithi187.stdout:(100/131): postgresql-server-10.21-2.module+el8 20 MB/s | 5.1 MB 00:00 2023-07-14T18:00:35.516 INFO:teuthology.orchestra.run.smithi187.stdout:(101/131): postgresql-contrib-10.21-2.module+el 4.3 MB/s | 808 kB 00:00 2023-07-14T18:00:35.558 INFO:teuthology.orchestra.run.smithi187.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 7.8 MB/s | 1.5 MB 00:00 2023-07-14T18:00:35.608 INFO:teuthology.orchestra.run.smithi187.stdout:(103/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 350 kB/s | 47 kB 00:00 2023-07-14T18:00:35.642 INFO:teuthology.orchestra.run.smithi187.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.6 MB/s | 200 kB 00:00 2023-07-14T18:00:35.692 INFO:teuthology.orchestra.run.smithi187.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 318 kB/s | 43 kB 00:00 2023-07-14T18:00:35.718 INFO:teuthology.orchestra.run.smithi187.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 638 kB/s | 48 kB 00:00 2023-07-14T18:00:35.743 INFO:teuthology.orchestra.run.smithi187.stdout:(107/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2023-07-14T18:00:35.793 INFO:teuthology.orchestra.run.smithi187.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 865 kB/s | 87 kB 00:00 2023-07-14T18:00:35.860 INFO:teuthology.orchestra.run.smithi187.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 932 kB/s | 109 kB 00:00 2023-07-14T18:00:35.877 INFO:teuthology.orchestra.run.smithi187.stdout:(110/131): python3-requests-oauthlib-1.0.0-1.el 268 kB/s | 43 kB 00:00 2023-07-14T18:00:35.944 INFO:teuthology.orchestra.run.smithi187.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.5 MB/s | 229 kB 00:00 2023-07-14T18:00:36.003 INFO:teuthology.orchestra.run.smithi187.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.8 MB/s | 560 kB 00:00 2023-07-14T18:00:36.019 INFO:teuthology.orchestra.run.smithi187.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 131 kB/s | 19 kB 00:00 2023-07-14T18:00:36.111 INFO:teuthology.orchestra.run.smithi187.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 926 kB/s | 155 kB 00:00 2023-07-14T18:00:36.145 INFO:teuthology.orchestra.run.smithi187.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.8 MB/s | 559 kB 00:00 2023-07-14T18:00:36.204 INFO:teuthology.orchestra.run.smithi187.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 97 kB/s | 18 kB 00:00 2023-07-14T18:00:36.329 INFO:teuthology.orchestra.run.smithi187.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 2.7 MB/s | 605 kB 00:00 2023-07-14T18:00:36.355 INFO:teuthology.orchestra.run.smithi187.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2023-07-14T18:00:36.413 INFO:teuthology.orchestra.run.smithi187.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 11 MB/s | 2.3 MB 00:00 2023-07-14T18:00:36.463 INFO:teuthology.orchestra.run.smithi187.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 541 kB/s | 72 kB 00:00 2023-07-14T18:00:36.530 INFO:teuthology.orchestra.run.smithi187.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 471 kB/s | 83 kB 00:00 2023-07-14T18:00:36.556 INFO:teuthology.orchestra.run.smithi187.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.0 MB/s | 98 kB 00:00 2023-07-14T18:00:36.581 INFO:teuthology.orchestra.run.smithi187.stdout:(123/131): elfutils-libelf-devel-0.186-1.el8.x8 357 kB/s | 60 kB 00:00 2023-07-14T18:00:36.665 INFO:teuthology.orchestra.run.smithi187.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 431 kB/s | 58 kB 00:00 2023-07-14T18:00:36.698 INFO:teuthology.orchestra.run.smithi187.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 5.4 MB/s | 644 kB 00:00 2023-07-14T18:00:36.724 INFO:teuthology.orchestra.run.smithi187.stdout:(126/131): libcom_err-devel-1.45.6-4.el8.x86_64 230 kB/s | 39 kB 00:00 2023-07-14T18:00:36.774 INFO:teuthology.orchestra.run.smithi187.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.5 MB/s | 171 kB 00:00 2023-07-14T18:00:36.832 INFO:teuthology.orchestra.run.smithi187.stdout:(128/131): NetworkManager-initscripts-updown-1. 1.0 MB/s | 138 kB 00:00 2023-07-14T18:00:36.991 INFO:teuthology.orchestra.run.smithi187.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.9 MB/s | 798 kB 00:00 2023-07-14T18:00:37.633 INFO:teuthology.orchestra.run.smithi187.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 4.0 MB/s | 3.7 MB 00:00 2023-07-14T18:00:38.244 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-07-14T18:00:38.525 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T18:00:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T18:00:38.900 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-14T18:00:39.062 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:00:39.200 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:00:39.418 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:00:39.821 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:00:39.822 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:00:39.822 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:00:39.837 INFO:teuthology.orchestra.run.smithi150.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T18:00:39.838 INFO:teuthology.orchestra.run.smithi150.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T18:00:39.839 INFO:teuthology.orchestra.run.smithi150.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T18:00:39.840 INFO:teuthology.orchestra.run.smithi150.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T18:00:39.843 INFO:teuthology.orchestra.run.smithi150.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T18:00:39.846 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T18:00:39.847 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T18:00:39.980 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-07-14T18:00:39.992 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================================================== 2023-07-14T18:00:39.993 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-07-14T18:00:39.993 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================================================== 2023-07-14T18:00:39.993 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-07-14T18:00:39.994 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T18:00:39.994 INFO:teuthology.orchestra.run.smithi150.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T18:00:39.994 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T18:00:39.994 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T18:00:39.994 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T18:00:39.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T18:00:39.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T18:00:39.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T18:00:39.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T18:00:39.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T18:00:39.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T18:00:39.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T18:00:39.996 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T18:00:39.996 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T18:00:39.996 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T18:00:39.996 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T18:00:39.996 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T18:00:39.996 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T18:00:39.996 INFO:teuthology.orchestra.run.smithi150.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T18:00:39.997 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T18:00:39.997 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T18:00:39.997 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T18:00:39.997 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T18:00:39.997 INFO:teuthology.orchestra.run.smithi150.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T18:00:39.997 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T18:00:39.997 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T18:00:39.998 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T18:00:39.998 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T18:00:39.998 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T18:00:39.998 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T18:00:39.998 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T18:00:39.998 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T18:00:39.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T18:00:39.999 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T18:00:39.999 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T18:00:39.999 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T18:00:39.999 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T18:00:39.999 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2023-07-14T18:00:39.999 INFO:teuthology.orchestra.run.smithi150.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T18:00:39.999 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T18:00:40.000 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T18:00:40.000 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-07-14T18:00:40.000 INFO:teuthology.orchestra.run.smithi150.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T18:00:40.000 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T18:00:40.000 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T18:00:40.001 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T18:00:40.001 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T18:00:40.001 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T18:00:40.001 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T18:00:40.002 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T18:00:40.002 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T18:00:40.002 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T18:00:40.002 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T18:00:40.002 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T18:00:40.002 INFO:teuthology.orchestra.run.smithi150.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T18:00:40.002 INFO:teuthology.orchestra.run.smithi150.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T18:00:40.003 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T18:00:40.003 INFO:teuthology.orchestra.run.smithi150.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T18:00:40.003 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T18:00:40.003 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T18:00:40.003 INFO:teuthology.orchestra.run.smithi150.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T18:00:40.003 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T18:00:40.003 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T18:00:40.004 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T18:00:40.004 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T18:00:40.004 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T18:00:40.004 INFO:teuthology.orchestra.run.smithi150.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T18:00:40.004 INFO:teuthology.orchestra.run.smithi150.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T18:00:40.004 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T18:00:40.004 INFO:teuthology.orchestra.run.smithi150.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T18:00:40.005 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T18:00:40.005 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T18:00:40.005 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T18:00:40.005 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T18:00:40.005 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T18:00:40.005 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T18:00:40.005 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T18:00:40.005 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T18:00:40.006 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T18:00:40.006 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T18:00:40.006 INFO:teuthology.orchestra.run.smithi150.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T18:00:40.006 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T18:00:40.006 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T18:00:40.006 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T18:00:40.006 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T18:00:40.007 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T18:00:40.007 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T18:00:40.007 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T18:00:40.007 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T18:00:40.007 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T18:00:40.007 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T18:00:40.007 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T18:00:40.007 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T18:00:40.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T18:00:40.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T18:00:40.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:00:40.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T18:00:40.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T18:00:40.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T18:00:40.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T18:00:40.009 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T18:00:40.009 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T18:00:40.009 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T18:00:40.009 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T18:00:40.009 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T18:00:40.009 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T18:00:40.009 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T18:00:40.010 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T18:00:40.010 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T18:00:40.010 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T18:00:40.010 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T18:00:40.010 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T18:00:40.010 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T18:00:40.010 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T18:00:40.010 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:00:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T18:00:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T18:00:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T18:00:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T18:00:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T18:00:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T18:00:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T18:00:40.012 INFO:teuthology.orchestra.run.smithi150.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T18:00:40.012 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T18:00:40.012 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T18:00:40.012 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T18:00:40.012 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T18:00:40.012 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T18:00:40.012 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T18:00:40.012 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T18:00:40.013 INFO:teuthology.orchestra.run.smithi150.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T18:00:40.013 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T18:00:40.013 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T18:00:40.013 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling module streams: 2023-07-14T18:00:40.013 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql 10 2023-07-14T18:00:40.013 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T18:00:40.013 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-07-14T18:00:40.014 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================================================== 2023-07-14T18:00:40.014 INFO:teuthology.orchestra.run.smithi150.stdout:Install 128 Packages 2023-07-14T18:00:40.014 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 3 Packages 2023-07-14T18:00:40.014 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T18:00:40.014 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 206 M 2023-07-14T18:00:40.015 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-07-14T18:00:40.035 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:00:40.429 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:00:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:00:40.709 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:00:40.812 INFO:teuthology.orchestra.run.smithi150.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 17 kB/s | 6.4 kB 00:00 2023-07-14T18:00:40.910 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:00:41.006 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:00:41.185 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:00:41.379 INFO:teuthology.orchestra.run.smithi150.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 1.5 MB/s | 850 kB 00:00 2023-07-14T18:00:41.404 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T18:00:41.488 INFO:teuthology.orchestra.run.smithi150.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 1.5 MB/s | 167 kB 00:00 2023-07-14T18:00:41.654 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T18:00:41.822 INFO:teuthology.orchestra.run.smithi150.stdout:(4/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 3.9 MB/s | 5.4 MB 00:01 2023-07-14T18:00:41.866 INFO:teuthology.orchestra.run.smithi187.stdout:(131/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 730 kB/s | 3.6 MB 00:05 2023-07-14T18:00:41.869 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:00:41.869 INFO:teuthology.orchestra.run.smithi187.stdout:Total 8.1 MB/s | 206 MB 00:25 2023-07-14T18:00:41.888 INFO:teuthology.orchestra.run.smithi150.stdout:(5/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 5.7 MB/s | 2.3 MB 00:00 2023-07-14T18:00:42.037 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T18:00:42.097 INFO:teuthology.orchestra.run.smithi150.stdout:(6/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 5.6 MB/s | 1.6 MB 00:00 2023-07-14T18:00:42.271 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T18:00:42.409 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:00:42.498 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:00:42.668 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:00:42.695 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-07-14T18:00:42.759 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:00:42.814 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-07-14T18:00:42.814 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-07-14T18:00:42.876 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:00:43.018 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:00:43.155 INFO:teuthology.orchestra.run.smithi150.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 3.6 MB/s | 4.6 MB 00:01 2023-07-14T18:00:43.409 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:00:43.531 INFO:teuthology.orchestra.run.smithi150.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 7.2 MB/s | 22 MB 00:03 2023-07-14T18:00:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 209 kB/s | 24 kB 00:00 2023-07-14T18:00:43.934 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-07-14T18:00:43.938 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-07-14T18:00:44.027 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:00:44.090 INFO:teuthology.orchestra.run.smithi150.stdout:(10/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 8.9 MB/s | 18 MB 00:01 2023-07-14T18:00:44.209 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:00:44.232 INFO:teuthology.orchestra.run.smithi150.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 1.8 MB/s | 260 kB 00:00 2023-07-14T18:00:44.340 INFO:teuthology.orchestra.run.smithi150.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 286 kB/s | 31 kB 00:00 2023-07-14T18:00:44.482 INFO:teuthology.orchestra.run.smithi150.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 5.3 MB/s | 772 kB 00:00 2023-07-14T18:00:44.624 INFO:teuthology.orchestra.run.smithi150.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 957 kB/s | 135 kB 00:00 2023-07-14T18:00:44.732 INFO:teuthology.orchestra.run.smithi150.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 1.3 MB/s | 139 kB 00:00 2023-07-14T18:00:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:00:44.858 INFO:teuthology.orchestra.run.smithi150.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 3.7 MB/s | 473 kB 00:00 2023-07-14T18:00:45.033 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:00:45.675 INFO:teuthology.orchestra.run.smithi150.stdout:(17/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 8.3 MB/s | 6.8 MB 00:00 2023-07-14T18:00:45.741 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:00:45.792 INFO:teuthology.orchestra.run.smithi150.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 384 kB/s | 45 kB 00:00 2023-07-14T18:00:45.917 INFO:teuthology.orchestra.run.smithi150.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 980 kB/s | 122 kB 00:00 2023-07-14T18:00:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-14T18:00:46.043 INFO:teuthology.orchestra.run.smithi150.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 1.5 MB/s | 197 kB 00:00 2023-07-14T18:00:46.193 INFO:teuthology.orchestra.run.smithi150.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 2.4 MB/s | 371 kB 00:00 2023-07-14T18:00:46.244 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-14T18:00:46.343 INFO:teuthology.orchestra.run.smithi150.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 2.4 MB/s | 364 kB 00:00 2023-07-14T18:00:46.462 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-14T18:00:46.468 INFO:teuthology.orchestra.run.smithi150.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 853 kB/s | 107 kB 00:00 2023-07-14T18:00:46.594 INFO:teuthology.orchestra.run.smithi150.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 751 kB/s | 94 kB 00:00 2023-07-14T18:00:46.689 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-14T18:00:46.897 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-14T18:00:47.052 INFO:teuthology.orchestra.run.smithi150.stdout:(25/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 7.7 MB/s | 3.5 MB 00:00 2023-07-14T18:00:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T18:00:47.195 INFO:teuthology.orchestra.run.smithi150.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.2 MB/s | 179 kB 00:00 2023-07-14T18:00:47.307 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T18:00:47.328 INFO:teuthology.orchestra.run.smithi150.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 173 kB/s | 23 kB 00:00 2023-07-14T18:00:47.444 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T18:00:47.553 INFO:teuthology.orchestra.run.smithi150.stdout:(28/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 3.1 MB/s | 14 MB 00:04 2023-07-14T18:00:47.579 INFO:teuthology.orchestra.run.smithi150.stdout:(29/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 516 kB/s | 129 kB 00:00 2023-07-14T18:00:48.371 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T18:00:48.429 INFO:teuthology.orchestra.run.smithi150.stdout:(30/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 10 MB/s | 48 MB 00:04 2023-07-14T18:00:48.471 INFO:teuthology.orchestra.run.smithi150.stdout:(31/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.9 MB/s | 1.7 MB 00:00 2023-07-14T18:00:48.554 INFO:teuthology.orchestra.run.smithi150.stdout:(32/131): ceph-mgr-modules-core-18.0.0-4940.g81 1.9 MB/s | 241 kB 00:00 2023-07-14T18:00:48.588 INFO:teuthology.orchestra.run.smithi150.stdout:(33/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 406 kB/s | 47 kB 00:00 2023-07-14T18:00:48.665 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-14T18:00:48.672 INFO:teuthology.orchestra.run.smithi150.stdout:(34/131): ceph-mgr-diskprediction-local-18.0.0- 6.8 MB/s | 7.4 MB 00:01 2023-07-14T18:00:48.697 INFO:teuthology.orchestra.run.smithi150.stdout:(35/131): ceph-prometheus-alerts-18.0.0-4940.g8 102 kB/s | 15 kB 00:00 2023-07-14T18:00:48.749 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-07-14T18:00:48.788 INFO:teuthology.orchestra.run.smithi150.stdout:(36/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 1.4 MB/s | 165 kB 00:00 2023-07-14T18:00:48.814 INFO:teuthology.orchestra.run.smithi150.stdout:(37/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 154 kB/s | 18 kB 00:00 2023-07-14T18:00:48.839 INFO:teuthology.orchestra.run.smithi150.stdout:(38/131): ceph-volume-18.0.0-4940.g81c9387d.el8 1.0 MB/s | 258 kB 00:00 2023-07-14T18:00:48.944 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T18:00:48.952 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-14T18:00:49.107 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T18:00:49.131 INFO:teuthology.orchestra.run.smithi150.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 185 kB/s | 59 kB 00:00 2023-07-14T18:00:49.156 INFO:teuthology.orchestra.run.smithi150.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 237 kB/s | 75 kB 00:00 2023-07-14T18:00:49.201 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-14T18:00:49.303 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-14T18:00:49.306 INFO:teuthology.orchestra.run.smithi150.stdout:(41/131): gperftools-libs-2.7-9.el8.x86_64.rpm 591 kB/s | 306 kB 00:00 2023-07-14T18:00:49.332 INFO:teuthology.orchestra.run.smithi150.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 249 kB/s | 44 kB 00:00 2023-07-14T18:00:49.391 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:00:49.457 INFO:teuthology.orchestra.run.smithi150.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.2 MB/s | 185 kB 00:00 2023-07-14T18:00:49.474 INFO:teuthology.orchestra.run.smithi150.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 232 kB/s | 33 kB 00:00 2023-07-14T18:00:49.496 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:00:49.575 INFO:teuthology.orchestra.run.smithi150.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 139 kB/s | 16 kB 00:00 2023-07-14T18:00:49.596 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:00:49.599 INFO:teuthology.orchestra.run.smithi150.stdout:(46/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.0 MB/s | 476 kB 00:00 2023-07-14T18:00:49.683 INFO:teuthology.orchestra.run.smithi150.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 826 kB/s | 173 kB 00:00 2023-07-14T18:00:49.734 INFO:teuthology.orchestra.run.smithi150.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 607 kB/s | 81 kB 00:00 2023-07-14T18:00:49.784 INFO:teuthology.orchestra.run.smithi150.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.8 MB/s | 384 kB 00:00 2023-07-14T18:00:49.801 INFO:teuthology.orchestra.run.smithi150.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 94 kB/s | 11 kB 00:00 2023-07-14T18:00:49.804 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:00:49.826 INFO:teuthology.orchestra.run.smithi150.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 191 kB/s | 18 kB 00:00 2023-07-14T18:00:49.893 INFO:teuthology.orchestra.run.smithi150.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 531 kB/s | 49 kB 00:00 2023-07-14T18:00:49.919 INFO:teuthology.orchestra.run.smithi150.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 637 kB/s | 59 kB 00:00 2023-07-14T18:00:49.994 INFO:teuthology.orchestra.run.smithi150.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 600 kB/s | 60 kB 00:00 2023-07-14T18:00:50.036 INFO:teuthology.orchestra.run.smithi150.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2023-07-14T18:00:50.086 INFO:teuthology.orchestra.run.smithi150.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2023-07-14T18:00:50.145 INFO:teuthology.orchestra.run.smithi150.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 310 kB/s | 34 kB 00:00 2023-07-14T18:00:50.186 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:00:50.195 INFO:teuthology.orchestra.run.smithi150.stdout:(58/131): python3-kubernetes-11.0.0-6.el8.noarc 2.8 MB/s | 1.2 MB 00:00 2023-07-14T18:00:50.254 INFO:teuthology.orchestra.run.smithi150.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 566 kB/s | 61 kB 00:00 2023-07-14T18:00:50.279 INFO:teuthology.orchestra.run.smithi150.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 1.0 MB/s | 196 kB 00:00 2023-07-14T18:00:50.304 INFO:teuthology.orchestra.run.smithi150.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 172 kB/s | 19 kB 00:00 2023-07-14T18:00:50.354 INFO:teuthology.orchestra.run.smithi150.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 238 kB/s | 24 kB 00:00 2023-07-14T18:00:50.388 INFO:teuthology.orchestra.run.smithi150.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 269 kB/s | 29 kB 00:00 2023-07-14T18:00:50.413 INFO:teuthology.orchestra.run.smithi150.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 253 kB/s | 27 kB 00:00 2023-07-14T18:00:50.421 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:00:50.480 INFO:teuthology.orchestra.run.smithi150.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.0 MB/s | 255 kB 00:00 2023-07-14T18:00:50.514 INFO:teuthology.orchestra.run.smithi150.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 610 kB/s | 61 kB 00:00 2023-07-14T18:00:50.589 INFO:teuthology.orchestra.run.smithi150.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 786 kB/s | 85 kB 00:00 2023-07-14T18:00:50.622 INFO:teuthology.orchestra.run.smithi150.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 211 kB/s | 23 kB 00:00 2023-07-14T18:00:50.648 INFO:teuthology.orchestra.run.smithi150.stdout:(69/131): python3-webob-1.8.5-1.el8.1.noarch.rp 967 kB/s | 251 kB 00:00 2023-07-14T18:00:50.673 INFO:teuthology.orchestra.run.smithi150.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-07-14T18:00:50.819 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:00:50.932 INFO:teuthology.orchestra.run.smithi150.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 435 kB/s | 110 kB 00:00 2023-07-14T18:00:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:00:51.107 INFO:teuthology.orchestra.run.smithi150.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 310 kB/s | 54 kB 00:00 2023-07-14T18:00:51.116 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:00:51.152 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:00:51.241 INFO:teuthology.orchestra.run.smithi150.stdout:(73/131): python3-scipy-1.0.0-21.module+el8.5.0 23 MB/s | 14 MB 00:00 2023-07-14T18:00:51.275 INFO:teuthology.orchestra.run.smithi150.stdout:(74/131): thrift-0.13.0-2.el8.x86_64.rpm 2.5 MB/s | 1.7 MB 00:00 2023-07-14T18:00:51.300 INFO:teuthology.orchestra.run.smithi150.stdout:(75/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2023-07-14T18:00:51.321 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:00:51.376 INFO:teuthology.orchestra.run.smithi150.stdout:(76/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2023-07-14T18:00:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:00:51.459 INFO:teuthology.orchestra.run.smithi150.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 4.4 MB/s | 688 kB 00:00 2023-07-14T18:00:51.484 INFO:teuthology.orchestra.run.smithi150.stdout:(78/131): python3-markupsafe-0.23-19.el8.x86_64 185 kB/s | 39 kB 00:00 2023-07-14T18:00:51.485 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-14T18:00:51.518 INFO:teuthology.orchestra.run.smithi150.stdout:(79/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.7 MB/s | 538 kB 00:00 2023-07-14T18:00:51.585 INFO:teuthology.orchestra.run.smithi150.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-07-14T18:00:51.588 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:00:51.610 INFO:teuthology.orchestra.run.smithi150.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-07-14T18:00:51.652 INFO:teuthology.orchestra.run.smithi150.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-07-14T18:00:51.678 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T18:00:51.702 INFO:teuthology.orchestra.run.smithi150.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 288 kB/s | 27 kB 00:00 2023-07-14T18:00:51.794 INFO:teuthology.orchestra.run.smithi150.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2023-07-14T18:00:51.809 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T18:00:51.828 INFO:teuthology.orchestra.run.smithi150.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 1.8 MB/s | 320 kB 00:00 2023-07-14T18:00:51.845 INFO:teuthology.orchestra.run.smithi150.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 309 kB/s | 44 kB 00:00 2023-07-14T18:00:51.901 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T18:00:51.978 INFO:teuthology.orchestra.run.smithi150.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 20 MB/s | 3.7 MB 00:00 2023-07-14T18:00:51.990 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T18:00:52.004 INFO:teuthology.orchestra.run.smithi150.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.4 MB/s | 225 kB 00:00 2023-07-14T18:00:52.029 INFO:teuthology.orchestra.run.smithi150.stdout:(89/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.2 MB/s | 457 kB 00:00 2023-07-14T18:00:52.080 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T18:00:52.104 INFO:teuthology.orchestra.run.smithi150.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 510 kB/s | 64 kB 00:00 2023-07-14T18:00:52.156 INFO:teuthology.orchestra.run.smithi150.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 839 kB/s | 126 kB 00:00 2023-07-14T18:00:52.205 INFO:teuthology.orchestra.run.smithi150.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 114 kB/s | 20 kB 00:00 2023-07-14T18:00:52.245 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T18:00:52.288 INFO:teuthology.orchestra.run.smithi150.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 563 kB/s | 103 kB 00:00 2023-07-14T18:00:52.338 INFO:teuthology.orchestra.run.smithi150.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 3.8 MB/s | 709 kB 00:00 2023-07-14T18:00:52.397 INFO:teuthology.orchestra.run.smithi150.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.8 MB/s | 198 kB 00:00 2023-07-14T18:00:52.409 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T18:00:52.448 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-14T18:00:52.489 INFO:teuthology.orchestra.run.smithi150.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 17 MB/s | 4.9 MB 00:00 2023-07-14T18:00:52.514 INFO:teuthology.orchestra.run.smithi150.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.7 MB/s | 202 kB 00:00 2023-07-14T18:00:52.615 INFO:teuthology.orchestra.run.smithi150.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-07-14T18:00:52.620 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-14T18:00:52.641 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T18:00:52.648 INFO:teuthology.orchestra.run.smithi150.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2023-07-14T18:00:52.757 INFO:teuthology.orchestra.run.smithi150.stdout:(100/131): postgresql-server-10.21-2.module+el8 21 MB/s | 5.1 MB 00:00 2023-07-14T18:00:52.779 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:00:52.790 INFO:teuthology.orchestra.run.smithi150.stdout:(101/131): postgresql-contrib-10.21-2.module+el 4.5 MB/s | 808 kB 00:00 2023-07-14T18:00:52.793 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-14T18:00:52.841 INFO:teuthology.orchestra.run.smithi150.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 7.8 MB/s | 1.5 MB 00:00 2023-07-14T18:00:52.883 INFO:teuthology.orchestra.run.smithi150.stdout:(103/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 373 kB/s | 47 kB 00:00 2023-07-14T18:00:52.894 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:00:52.908 INFO:teuthology.orchestra.run.smithi150.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.7 MB/s | 200 kB 00:00 2023-07-14T18:00:52.927 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-14T18:00:52.950 INFO:teuthology.orchestra.run.smithi150.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 390 kB/s | 43 kB 00:00 2023-07-14T18:00:53.001 INFO:teuthology.orchestra.run.smithi150.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.8 MB/s | 220 kB 00:00 2023-07-14T18:00:53.071 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:00:53.101 INFO:teuthology.orchestra.run.smithi150.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 250 kB/s | 48 kB 00:00 2023-07-14T18:00:53.126 INFO:teuthology.orchestra.run.smithi150.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 341 kB/s | 43 kB 00:00 2023-07-14T18:00:53.138 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:00:53.185 INFO:teuthology.orchestra.run.smithi150.stdout:(109/131): libsepol-devel-2.9-3.el8.x86_64.rpm 371 kB/s | 87 kB 00:00 2023-07-14T18:00:53.191 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-14T18:00:53.227 INFO:teuthology.orchestra.run.smithi150.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 868 kB/s | 109 kB 00:00 2023-07-14T18:00:53.262 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:00:53.285 INFO:teuthology.orchestra.run.smithi150.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.4 MB/s | 229 kB 00:00 2023-07-14T18:00:53.336 INFO:teuthology.orchestra.run.smithi150.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.6 MB/s | 560 kB 00:00 2023-07-14T18:00:53.369 INFO:teuthology.orchestra.run.smithi150.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 131 kB/s | 19 kB 00:00 2023-07-14T18:00:53.385 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-14T18:00:53.386 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:00:53.419 INFO:teuthology.orchestra.run.smithi150.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.1 MB/s | 155 kB 00:00 2023-07-14T18:00:53.478 INFO:teuthology.orchestra.run.smithi150.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.8 MB/s | 559 kB 00:00 2023-07-14T18:00:53.503 INFO:teuthology.orchestra.run.smithi150.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 134 kB/s | 18 kB 00:00 2023-07-14T18:00:53.529 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:00:53.554 INFO:teuthology.orchestra.run.smithi150.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.4 MB/s | 605 kB 00:00 2023-07-14T18:00:53.604 INFO:teuthology.orchestra.run.smithi150.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.8 MB/s | 238 kB 00:00 2023-07-14T18:00:53.646 INFO:teuthology.orchestra.run.smithi150.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 16 MB/s | 2.3 MB 00:00 2023-07-14T18:00:53.671 INFO:teuthology.orchestra.run.smithi150.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 616 kB/s | 72 kB 00:00 2023-07-14T18:00:53.721 INFO:teuthology.orchestra.run.smithi150.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 706 kB/s | 83 kB 00:00 2023-07-14T18:00:53.778 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-14T18:00:53.780 INFO:teuthology.orchestra.run.smithi150.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 447 kB/s | 60 kB 00:00 2023-07-14T18:00:53.806 INFO:teuthology.orchestra.run.smithi150.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 733 kB/s | 98 kB 00:00 2023-07-14T18:00:53.856 INFO:teuthology.orchestra.run.smithi150.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 430 kB/s | 58 kB 00:00 2023-07-14T18:00:53.939 INFO:teuthology.orchestra.run.smithi150.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 243 kB/s | 39 kB 00:00 2023-07-14T18:00:53.964 INFO:teuthology.orchestra.run.smithi150.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.0 MB/s | 644 kB 00:00 2023-07-14T18:00:53.990 INFO:teuthology.orchestra.run.smithi150.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.2 MB/s | 171 kB 00:00 2023-07-14T18:00:54.035 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-14T18:00:54.090 INFO:teuthology.orchestra.run.smithi150.stdout:(128/131): NetworkManager-initscripts-updown-1. 913 kB/s | 138 kB 00:00 2023-07-14T18:00:54.132 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:00:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-14T18:00:54.274 INFO:teuthology.orchestra.run.smithi150.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.3 MB/s | 798 kB 00:00 2023-07-14T18:00:54.295 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:00:54.488 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-14T18:00:54.771 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-14T18:00:54.890 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:00:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:00:55.100 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-14T18:00:55.107 INFO:teuthology.orchestra.run.smithi150.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 3.2 MB/s | 3.6 MB 00:01 2023-07-14T18:00:55.340 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-14T18:00:55.524 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-14T18:00:55.724 INFO:teuthology.orchestra.run.smithi150.stdout:(131/131): librados2-18.0.0-4940.g81c9387d.el8. 2.1 MB/s | 3.7 MB 00:01 2023-07-14T18:00:55.727 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:00:55.727 INFO:teuthology.orchestra.run.smithi150.stdout:Total 13 MB/s | 206 MB 00:15 2023-07-14T18:00:55.752 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-14T18:00:55.756 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:00:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-14T18:00:55.993 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-14T18:00:56.113 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-14T18:00:56.214 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-14T18:00:56.406 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-14T18:00:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-14T18:00:56.539 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-07-14T18:00:56.626 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-14T18:00:56.660 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-07-14T18:00:56.660 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-07-14T18:00:56.687 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-14T18:00:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-14T18:00:56.928 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-14T18:00:57.073 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T18:00:57.189 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-14T18:00:57.244 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T18:00:57.347 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-14T18:00:57.373 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T18:00:57.510 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-14T18:00:57.706 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:00:57.760 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:00:57.764 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-07-14T18:00:57.768 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-07-14T18:00:58.009 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-14T18:00:58.235 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-14T18:00:58.323 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T18:00:58.390 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:00:58.471 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:00:58.634 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-14T18:00:58.678 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-14T18:00:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-14T18:00:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-14T18:00:59.031 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T18:00:59.135 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-14T18:00:59.210 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T18:00:59.345 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:00:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T18:00:59.713 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T18:00:59.894 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T18:01:00.051 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T18:01:00.250 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T18:01:00.470 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T18:01:00.651 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T18:01:00.817 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T18:01:00.954 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T18:01:01.106 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:01:01.194 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T18:01:01.419 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T18:01:01.429 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-14T18:01:01.658 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T18:01:01.671 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T18:01:01.846 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T18:01:01.911 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T18:01:02.059 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T18:01:02.122 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T18:01:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T18:01:02.321 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T18:01:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-14T18:01:02.502 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-07-14T18:01:02.547 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T18:01:02.555 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-14T18:01:02.708 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T18:01:02.747 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-14T18:01:02.870 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T18:01:02.915 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-14T18:01:03.075 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-14T18:01:03.144 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-14T18:01:03.254 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:01:03.359 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-14T18:01:03.367 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:01:03.610 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:01:03.653 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T18:01:03.733 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-14T18:01:03.950 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T18:01:04.012 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:01:04.014 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-14T18:01:04.218 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:01:04.255 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T18:01:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T18:01:04.574 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-14T18:01:04.644 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:01:04.800 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-14T18:01:04.831 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T18:01:04.843 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:01:04.967 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:01:05.022 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T18:01:05.025 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-14T18:01:05.160 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:01:05.256 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:01:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-14T18:01:05.435 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:01:05.552 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-14T18:01:05.680 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T18:01:05.737 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-14T18:01:05.832 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T18:01:05.896 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T18:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-14T18:01:05.988 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:01:06.065 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:01:06.140 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-14T18:01:06.288 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T18:01:06.297 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-14T18:01:06.318 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T18:01:06.555 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T18:01:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-14T18:01:06.692 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:01:06.809 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:01:06.875 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-14T18:01:06.968 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:01:07.043 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:01:07.107 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-14T18:01:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-14T18:01:07.543 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-14T18:01:07.626 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:01:07.687 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-14T18:01:07.753 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:01:07.860 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:01:07.941 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:01:07.951 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:01:08.191 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-14T18:01:08.384 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-14T18:01:08.535 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:01:08.552 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:01:08.619 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:01:08.759 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:01:08.840 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-14T18:01:08.986 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-14T18:01:09.176 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T18:01:09.347 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T18:01:09.360 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:01:09.584 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:01:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T18:01:09.834 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T18:01:09.961 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:01:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T18:01:10.155 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:01:10.171 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T18:01:10.281 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:01:10.361 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T18:01:10.532 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-14T18:01:10.548 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T18:01:10.722 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T18:01:10.728 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:01:10.744 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-14T18:01:10.905 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T18:01:10.963 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-14T18:01:11.018 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:01:11.033 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T18:01:11.092 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:01:11.157 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-14T18:01:11.221 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:01:11.240 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T18:01:11.390 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-14T18:01:11.464 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T18:01:11.628 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T18:01:11.649 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T18:01:11.783 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T18:01:11.904 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T18:01:11.906 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T18:01:12.134 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T18:01:12.309 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T18:01:12.518 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T18:01:12.831 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T18:01:13.166 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-14T18:01:13.437 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-14T18:01:13.618 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T18:01:13.685 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-14T18:01:13.875 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:01:13.896 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T18:01:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T18:01:14.442 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T18:01:14.727 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T18:01:14.985 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T18:01:15.654 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:01:15.769 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T18:01:15.926 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:01:15.988 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:01:16.002 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-14T18:01:16.143 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T18:01:16.220 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T18:01:16.394 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T18:01:16.606 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T18:01:16.787 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T18:01:16.949 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-14T18:01:17.110 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-14T18:01:17.303 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-14T18:01:17.446 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-14T18:01:17.682 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-14T18:01:17.865 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-14T18:01:18.242 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-14T18:01:18.537 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-14T18:01:18.744 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-14T18:01:18.971 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-14T18:01:19.222 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-14T18:01:19.542 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-14T18:01:19.741 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:01:19.766 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-14T18:01:19.984 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-14T18:01:20.107 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:01:20.213 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-14T18:01:20.387 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-14T18:01:20.553 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-14T18:01:20.697 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:01:20.850 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-14T18:01:20.988 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:01:21.074 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:01:21.104 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-14T18:01:21.208 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:01:21.354 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-14T18:01:21.573 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-14T18:01:21.749 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-14T18:01:21.884 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-14T18:01:22.058 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:01:22.134 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:01:22.389 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-14T18:01:22.570 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-14T18:01:22.733 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:01:22.795 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:01:23.037 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-14T18:01:23.192 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-14T18:01:23.383 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T18:01:23.580 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T18:01:23.824 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T18:01:24.032 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T18:01:24.171 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T18:01:24.319 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T18:01:24.536 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T18:01:24.763 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T18:01:24.953 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T18:01:25.169 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T18:01:25.314 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T18:01:25.530 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T18:01:25.778 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T18:01:25.955 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T18:01:26.206 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T18:01:26.399 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T18:01:26.566 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T18:01:26.776 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T18:01:27.937 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T18:01:28.245 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T18:01:28.482 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T18:01:28.852 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T18:01:29.124 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T18:01:29.341 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T18:01:30.135 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T18:01:30.142 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:01:30.142 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2023-07-14T18:01:30.143 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2023-07-14T18:01:30.143 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2023-07-14T18:01:30.143 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /run 2023-07-14T18:01:30.143 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2023-07-14T18:01:30.143 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2023-07-14T18:01:30.144 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2023-07-14T18:01:30.144 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2023-07-14T18:01:30.144 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2023-07-14T18:01:30.144 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:01:30.324 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:01:30.400 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:01:30.518 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:01:30.583 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T18:01:32.096 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:01:32.281 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:01:32.499 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:01:32.667 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:01:33.121 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:01:33.309 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:01:33.326 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:01:34.225 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:01:34.441 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:01:34.962 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:01:35.025 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:01:35.133 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:01:35.256 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T18:01:35.295 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:01:35.380 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:01:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T18:01:35.591 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:01:35.666 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:01:35.880 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:01:36.180 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:01:37.136 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:01:37.299 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:01:37.389 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:01:37.741 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T18:01:37.960 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:01:37.996 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:01:38.141 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:01:38.152 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:01:39.493 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T18:01:39.713 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:01:39.747 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:01:39.907 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:01:39.949 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:01:40.124 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:01:40.476 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:01:40.608 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:01:40.608 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2023-07-14T18:01:40.608 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2023-07-14T18:01:40.609 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2023-07-14T18:01:40.609 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /run 2023-07-14T18:01:40.609 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2023-07-14T18:01:40.609 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2023-07-14T18:01:40.609 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2023-07-14T18:01:40.609 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2023-07-14T18:01:40.609 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /dev 2023-07-14T18:01:40.610 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T18:01:40.682 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:01:40.948 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:01:42.569 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:01:42.759 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:01:42.978 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:01:43.164 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:01:43.618 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:01:43.798 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:01:43.815 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:01:45.233 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:01:45.512 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T18:01:45.590 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:01:45.706 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:01:45.728 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:01:45.836 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T18:01:45.895 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:01:46.044 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T18:01:46.230 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:01:46.446 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:01:46.733 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:01:46.767 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-14T18:01:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-14T18:01:47.309 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 119/134 2023-07-14T18:01:47.510 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:01:47.684 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T18:01:47.693 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:01:47.832 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:01:47.880 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:01:47.970 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:01:48.336 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T18:01:48.568 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:01:48.603 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:01:48.788 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:01:48.799 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:01:50.011 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:01:50.084 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T18:01:50.266 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:01:50.272 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:01:50.290 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:01:50.300 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:01:50.488 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:01:50.530 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:01:50.554 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T18:01:50.696 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:01:50.746 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T18:01:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:01:51.082 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T18:01:51.205 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:01:51.415 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T18:01:51.572 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:01:51.581 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:01:51.771 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-14T18:01:51.930 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:01:52.016 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:01:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:01:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:01:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:01:52.450 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:01:52.450 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:01:52.728 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:01:52.962 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:01:53.501 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:01:53.502 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T18:01:53.502 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T18:01:53.502 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T18:01:53.502 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T18:01:53.502 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T18:01:53.503 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:01:53.503 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:01:53.503 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:01:53.503 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:01:53.503 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T18:01:53.503 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T18:01:53.503 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T18:01:53.503 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T18:01:53.504 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T18:01:53.504 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T18:01:53.504 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T18:01:53.504 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T18:01:53.504 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T18:01:53.504 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T18:01:53.504 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T18:01:53.505 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T18:01:53.506 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T18:01:53.506 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T18:01:53.506 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T18:01:53.506 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T18:01:53.506 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T18:01:53.506 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T18:01:53.506 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T18:01:53.507 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T18:01:53.507 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T18:01:53.513 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T18:01:53.514 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T18:01:53.514 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T18:01:53.514 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T18:01:53.514 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T18:01:53.514 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T18:01:53.514 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T18:01:53.514 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T18:01:53.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T18:01:53.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T18:01:53.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T18:01:53.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T18:01:53.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T18:01:53.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T18:01:53.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T18:01:53.515 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T18:01:53.516 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T18:01:53.516 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T18:01:53.516 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T18:01:53.516 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T18:01:53.516 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T18:01:53.516 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T18:01:53.517 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T18:01:53.517 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T18:01:53.517 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T18:01:53.517 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T18:01:53.517 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T18:01:53.517 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T18:01:53.517 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T18:01:53.517 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T18:01:53.518 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T18:01:53.518 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T18:01:53.518 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T18:01:53.518 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T18:01:53.518 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T18:01:53.518 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T18:01:53.518 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T18:01:53.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T18:01:53.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T18:01:53.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T18:01:53.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T18:01:53.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-14T18:01:53.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-14T18:01:53.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-14T18:01:53.519 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-14T18:01:53.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-14T18:01:53.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-14T18:01:53.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-14T18:01:53.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-14T18:01:53.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-14T18:01:53.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-14T18:01:53.520 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-14T18:01:53.521 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-14T18:01:53.521 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-14T18:01:53.521 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-14T18:01:53.521 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-14T18:01:53.521 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-14T18:01:53.521 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-14T18:01:53.521 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-14T18:01:53.522 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-14T18:01:53.522 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-14T18:01:53.522 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-14T18:01:53.522 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-14T18:01:53.522 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-14T18:01:53.522 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-14T18:01:53.522 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-14T18:01:53.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-14T18:01:53.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-14T18:01:53.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-14T18:01:53.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-14T18:01:53.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-14T18:01:53.523 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-14T18:01:53.524 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-14T18:01:53.524 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-14T18:01:53.524 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-14T18:01:53.524 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-14T18:01:53.524 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-14T18:01:53.525 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-14T18:01:53.525 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-14T18:01:53.525 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-14T18:01:53.525 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-14T18:01:53.526 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-14T18:01:53.526 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-14T18:01:53.526 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-14T18:01:53.527 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-14T18:01:53.527 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-14T18:01:53.527 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-14T18:01:53.527 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-14T18:01:53.528 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-14T18:01:53.528 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:01:53.528 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-14T18:01:53.529 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:01:53.529 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-14T18:01:53.529 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-14T18:01:53.529 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-14T18:01:53.530 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-14T18:01:53.530 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-14T18:01:53.530 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:01:53.530 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T18:01:53.531 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T18:01:53.531 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T18:01:53.531 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:01:53.531 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T18:01:54.188 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:01:54.189 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2023-07-14T18:01:54.189 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2023-07-14T18:01:54.189 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /dev 2023-07-14T18:01:54.189 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /run 2023-07-14T18:01:54.189 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2023-07-14T18:01:54.189 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2023-07-14T18:01:54.190 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2023-07-14T18:01:54.190 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2023-07-14T18:01:54.190 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /dev 2023-07-14T18:01:54.190 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T18:01:54.547 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:01:55.744 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:01:55.974 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:01:55.975 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:52 ago on Fri 14 Jul 2023 06:00:03 PM UTC. 2023-07-14T18:01:56.044 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T18:01:56.130 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:01:56.195 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:01:56.318 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:01:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:01:56.537 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:01:56.747 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:01:57.202 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:01:57.261 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-14T18:01:57.364 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:01:57.381 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:01:57.585 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-14T18:01:57.790 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 119/134 2023-07-14T18:01:57.974 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:01:58.132 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T18:01:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:01:59.033 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:01:59.212 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:01:59.337 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T18:01:59.520 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T18:01:59.688 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:01:59.905 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:02:00.209 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:02:00.359 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:02:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:02:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:02:00.829 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T18:02:01.102 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T18:02:01.171 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:02:01.330 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:02:01.420 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:02:01.442 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T18:02:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T18:02:01.780 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T18:02:01.844 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:02:01.853 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:02:02.011 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-14T18:02:02.022 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:02:02.057 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:02:02.213 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:02:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:02:02.224 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:02:02.303 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:02:02.503 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:02:02.515 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:02:02.515 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:02:02.714 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:02:02.714 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:02:02.962 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:02:03.193 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:02:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T18:02:03.722 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:02:03.722 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T18:02:03.722 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T18:02:03.722 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T18:02:03.722 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T18:02:03.722 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T18:02:03.723 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:02:03.723 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:02:03.723 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:02:03.723 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:02:03.723 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T18:02:03.723 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T18:02:03.723 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T18:02:03.724 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T18:02:03.724 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T18:02:03.724 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T18:02:03.724 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T18:02:03.724 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T18:02:03.724 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T18:02:03.724 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T18:02:03.724 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T18:02:03.725 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T18:02:03.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T18:02:03.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T18:02:03.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T18:02:03.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T18:02:03.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T18:02:03.726 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T18:02:03.727 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T18:02:03.727 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T18:02:03.727 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T18:02:03.727 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T18:02:03.736 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T18:02:03.736 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T18:02:03.736 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T18:02:03.736 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T18:02:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T18:02:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T18:02:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T18:02:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T18:02:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T18:02:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T18:02:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T18:02:03.737 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T18:02:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T18:02:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T18:02:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T18:02:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T18:02:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T18:02:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T18:02:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T18:02:03.738 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T18:02:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T18:02:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T18:02:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T18:02:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T18:02:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T18:02:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T18:02:03.739 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T18:02:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T18:02:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T18:02:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T18:02:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T18:02:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T18:02:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T18:02:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T18:02:03.740 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T18:02:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T18:02:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T18:02:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T18:02:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T18:02:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T18:02:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-14T18:02:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-14T18:02:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-14T18:02:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-14T18:02:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-14T18:02:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-14T18:02:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-14T18:02:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-14T18:02:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-14T18:02:03.742 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-14T18:02:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-14T18:02:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-14T18:02:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-14T18:02:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-14T18:02:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-14T18:02:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-14T18:02:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-14T18:02:03.743 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-14T18:02:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-14T18:02:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-14T18:02:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-14T18:02:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-14T18:02:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-14T18:02:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-14T18:02:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-14T18:02:03.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-14T18:02:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-14T18:02:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-14T18:02:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-14T18:02:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-14T18:02:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-14T18:02:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-14T18:02:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-14T18:02:03.745 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-14T18:02:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-14T18:02:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-14T18:02:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-14T18:02:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-14T18:02:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-14T18:02:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-14T18:02:03.746 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-14T18:02:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-14T18:02:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-14T18:02:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-14T18:02:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-14T18:02:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-14T18:02:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-14T18:02:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-14T18:02:03.747 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:02:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-14T18:02:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:02:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-14T18:02:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-14T18:02:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-14T18:02:03.748 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-14T18:02:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-14T18:02:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:02:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T18:02:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T18:02:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T18:02:03.749 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:02:03.750 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T18:02:03.760 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:02:03.794 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:02:03.947 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:02:03.989 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:02:04.053 INFO:teuthology.orchestra.run.smithi115.stdout:Installed products updated. 2023-07-14T18:02:04.163 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:02:04.267 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:02:04.268 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2023-07-14T18:02:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T18:02:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.268 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.268 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-07-14T18:02:04.269 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T18:02:04.269 INFO:teuthology.orchestra.run.smithi115.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T18:02:04.269 INFO:teuthology.orchestra.run.smithi115.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T18:02:04.269 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.269 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.269 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.269 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.269 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.270 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.270 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.270 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.270 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.270 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.270 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.270 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.271 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.271 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.271 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.271 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.271 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.271 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.271 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.271 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.272 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.272 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.272 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:04.272 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T18:02:04.272 INFO:teuthology.orchestra.run.smithi115.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T18:02:04.272 INFO:teuthology.orchestra.run.smithi115.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T18:02:04.272 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T18:02:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: jq-1.6-3.el8.x86_64 2023-07-14T18:02:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T18:02:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T18:02:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T18:02:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.273 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T18:02:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:02:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T18:02:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T18:02:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:02:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T18:02:04.274 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.275 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.275 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T18:02:04.275 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.275 INFO:teuthology.orchestra.run.smithi115.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T18:02:04.275 INFO:teuthology.orchestra.run.smithi115.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T18:02:04.275 INFO:teuthology.orchestra.run.smithi115.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T18:02:04.275 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T18:02:04.276 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T18:02:04.276 INFO:teuthology.orchestra.run.smithi115.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T18:02:04.276 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T18:02:04.276 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T18:02:04.276 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T18:02:04.276 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T18:02:04.276 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T18:02:04.276 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T18:02:04.277 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T18:02:04.277 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T18:02:04.277 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:02:04.277 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:02:04.277 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:02:04.277 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T18:02:04.277 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T18:02:04.278 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T18:02:04.278 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T18:02:04.278 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T18:02:04.278 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.278 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.278 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.278 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T18:02:04.278 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T18:02:04.279 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T18:02:04.279 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T18:02:04.279 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T18:02:04.279 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T18:02:04.279 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T18:02:04.279 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T18:02:04.279 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T18:02:04.279 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T18:02:04.280 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T18:02:04.280 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T18:02:04.280 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T18:02:04.280 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T18:02:04.280 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T18:02:04.280 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T18:02:04.280 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T18:02:04.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T18:02:04.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T18:02:04.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T18:02:04.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T18:02:04.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T18:02:04.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T18:02:04.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T18:02:04.281 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T18:02:04.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T18:02:04.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T18:02:04.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T18:02:04.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T18:02:04.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T18:02:04.282 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T18:02:04.283 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.283 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T18:02:04.283 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T18:02:04.283 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T18:02:04.283 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T18:02:04.283 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T18:02:04.283 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T18:02:04.283 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T18:02:04.284 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T18:02:04.284 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T18:02:04.284 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T18:02:04.284 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T18:02:04.284 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T18:02:04.284 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T18:02:04.284 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.284 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.285 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:04.285 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T18:02:04.285 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T18:02:04.285 INFO:teuthology.orchestra.run.smithi115.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T18:02:04.285 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T18:02:04.285 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T18:02:04.286 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T18:02:04.286 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:02:04.286 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-07-14T18:02:04.459 DEBUG:teuthology.parallel:result is None 2023-07-14T18:02:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:02:04.671 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:02:06.187 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:02:06.187 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:02 ago on Fri 14 Jul 2023 06:00:04 PM UTC. 2023-07-14T18:02:09.219 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:02:09.518 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T18:02:09.679 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:02:09.879 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:02:10.793 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-14T18:02:11.127 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-14T18:02:11.307 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 119/134 2023-07-14T18:02:11.491 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:02:11.674 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T18:02:11.838 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:02:13.993 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:02:14.254 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:02:14.272 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:02:14.421 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T18:02:14.712 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T18:02:15.024 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T18:02:15.305 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T18:02:15.437 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:02:15.445 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:02:15.615 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-14T18:02:15.853 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:02:15.934 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:02:16.135 INFO:teuthology.orchestra.run.smithi187.stdout:Installed products updated. 2023-07-14T18:02:16.153 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:02:16.165 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:02:16.166 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:02:16.354 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T18:02:16.354 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2023-07-14T18:02:16.354 INFO:teuthology.orchestra.run.smithi187.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T18:02:16.354 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.354 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.355 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-07-14T18:02:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T18:02:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T18:02:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T18:02:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.355 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.356 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.357 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T18:02:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T18:02:16.358 INFO:teuthology.orchestra.run.smithi187.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T18:02:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T18:02:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: jq-1.6-3.el8.x86_64 2023-07-14T18:02:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T18:02:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T18:02:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T18:02:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.359 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.360 INFO:teuthology.orchestra.run.smithi187.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T18:02:16.360 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:02:16.360 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T18:02:16.360 INFO:teuthology.orchestra.run.smithi187.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T18:02:16.360 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:02:16.360 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T18:02:16.360 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T18:02:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T18:02:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T18:02:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T18:02:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T18:02:16.361 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T18:02:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T18:02:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T18:02:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T18:02:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T18:02:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T18:02:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T18:02:16.362 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T18:02:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T18:02:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T18:02:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:02:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:02:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:02:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T18:02:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T18:02:16.363 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T18:02:16.364 INFO:teuthology.orchestra.run.smithi187.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T18:02:16.364 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T18:02:16.364 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.364 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.364 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.364 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T18:02:16.364 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T18:02:16.364 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T18:02:16.365 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T18:02:16.365 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T18:02:16.365 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T18:02:16.365 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T18:02:16.365 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T18:02:16.365 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T18:02:16.365 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T18:02:16.365 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T18:02:16.366 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T18:02:16.366 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T18:02:16.366 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T18:02:16.366 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T18:02:16.366 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T18:02:16.366 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T18:02:16.366 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T18:02:16.366 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T18:02:16.367 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T18:02:16.367 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T18:02:16.367 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T18:02:16.367 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T18:02:16.367 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T18:02:16.367 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T18:02:16.367 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T18:02:16.368 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T18:02:16.368 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T18:02:16.368 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T18:02:16.368 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.368 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.368 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T18:02:16.368 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T18:02:16.368 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T18:02:16.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T18:02:16.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T18:02:16.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T18:02:16.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T18:02:16.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T18:02:16.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T18:02:16.369 INFO:teuthology.orchestra.run.smithi187.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T18:02:16.370 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T18:02:16.370 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T18:02:16.370 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T18:02:16.370 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T18:02:16.370 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T18:02:16.370 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.370 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.371 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:16.371 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T18:02:16.371 INFO:teuthology.orchestra.run.smithi187.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T18:02:16.371 INFO:teuthology.orchestra.run.smithi187.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T18:02:16.371 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T18:02:16.371 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T18:02:16.372 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T18:02:16.372 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T18:02:16.372 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-07-14T18:02:16.372 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:02:16.372 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:02:16.607 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:02:16.819 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:02:16.900 DEBUG:teuthology.parallel:result is None 2023-07-14T18:02:17.335 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:02:17.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T18:02:17.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T18:02:17.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T18:02:17.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T18:02:17.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T18:02:17.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:02:17.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:02:17.337 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:02:17.337 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:02:17.337 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T18:02:17.337 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T18:02:17.337 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T18:02:17.337 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T18:02:17.337 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T18:02:17.338 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T18:02:17.338 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T18:02:17.338 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T18:02:17.338 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T18:02:17.338 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T18:02:17.338 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T18:02:17.338 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T18:02:17.339 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T18:02:17.339 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T18:02:17.339 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T18:02:17.339 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T18:02:17.346 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T18:02:17.346 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T18:02:17.346 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T18:02:17.346 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T18:02:17.346 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T18:02:17.347 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T18:02:17.347 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T18:02:17.347 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T18:02:17.347 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T18:02:17.347 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T18:02:17.347 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T18:02:17.347 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T18:02:17.348 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T18:02:17.348 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T18:02:17.348 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T18:02:17.348 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T18:02:17.348 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T18:02:17.348 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T18:02:17.348 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T18:02:17.349 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T18:02:17.349 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T18:02:17.349 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T18:02:17.349 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T18:02:17.349 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T18:02:17.349 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T18:02:17.349 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T18:02:17.349 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T18:02:17.350 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T18:02:17.350 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T18:02:17.350 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T18:02:17.350 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T18:02:17.350 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T18:02:17.350 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T18:02:17.350 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T18:02:17.351 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T18:02:17.351 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T18:02:17.351 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T18:02:17.351 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T18:02:17.351 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T18:02:17.351 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T18:02:17.351 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T18:02:17.351 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T18:02:17.352 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T18:02:17.352 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T18:02:17.352 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T18:02:17.352 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T18:02:17.352 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-14T18:02:17.352 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-14T18:02:17.352 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-14T18:02:17.353 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-14T18:02:17.353 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-14T18:02:17.353 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-14T18:02:17.353 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-14T18:02:17.353 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-14T18:02:17.353 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-14T18:02:17.353 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-14T18:02:17.353 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-14T18:02:17.354 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-14T18:02:17.354 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-14T18:02:17.354 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-14T18:02:17.354 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-14T18:02:17.354 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-14T18:02:17.354 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-14T18:02:17.354 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-14T18:02:17.354 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-14T18:02:17.355 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-14T18:02:17.355 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-14T18:02:17.355 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-14T18:02:17.355 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-14T18:02:17.355 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-14T18:02:17.355 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-14T18:02:17.355 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-14T18:02:17.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-14T18:02:17.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-14T18:02:17.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-14T18:02:17.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-14T18:02:17.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-14T18:02:17.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-14T18:02:17.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-14T18:02:17.357 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-14T18:02:17.357 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-14T18:02:17.357 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-14T18:02:17.357 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-14T18:02:17.357 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-14T18:02:17.357 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-14T18:02:17.358 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-14T18:02:17.358 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-14T18:02:17.358 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-14T18:02:17.359 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-14T18:02:17.359 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-14T18:02:17.360 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-14T18:02:17.360 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-14T18:02:17.360 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-14T18:02:17.361 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-14T18:02:17.361 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:02:17.361 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-14T18:02:17.361 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:02:17.362 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-14T18:02:17.362 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-14T18:02:17.362 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-14T18:02:17.363 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-14T18:02:17.363 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-14T18:02:17.363 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:02:17.363 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T18:02:17.364 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T18:02:17.364 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T18:02:17.364 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:02:17.365 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T18:02:19.895 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:02:19.896 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:51 ago on Fri 14 Jul 2023 06:00:28 PM UTC. 2023-07-14T18:02:28.557 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-07-14T18:02:28.777 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T18:02:28.777 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2023-07-14T18:02:28.777 INFO:teuthology.orchestra.run.smithi150.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T18:02:28.778 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.778 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.778 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-07-14T18:02:28.778 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T18:02:28.778 INFO:teuthology.orchestra.run.smithi150.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T18:02:28.778 INFO:teuthology.orchestra.run.smithi150.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T18:02:28.778 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.778 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.779 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.779 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.779 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.779 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.779 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.779 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.779 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.780 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.780 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.780 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.780 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.780 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.780 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.780 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.781 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.781 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.781 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.781 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.781 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.781 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.781 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:02:28.782 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T18:02:28.782 INFO:teuthology.orchestra.run.smithi150.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T18:02:28.782 INFO:teuthology.orchestra.run.smithi150.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T18:02:28.782 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T18:02:28.782 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-3.el8.x86_64 2023-07-14T18:02:28.782 INFO:teuthology.orchestra.run.smithi150.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T18:02:28.782 INFO:teuthology.orchestra.run.smithi150.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T18:02:28.783 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T18:02:28.783 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.783 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.783 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.783 INFO:teuthology.orchestra.run.smithi150.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T18:02:28.783 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:02:28.783 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T18:02:28.783 INFO:teuthology.orchestra.run.smithi150.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T18:02:28.784 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:02:28.784 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T18:02:28.784 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.784 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.784 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T18:02:28.784 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.784 INFO:teuthology.orchestra.run.smithi150.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T18:02:28.785 INFO:teuthology.orchestra.run.smithi150.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T18:02:28.785 INFO:teuthology.orchestra.run.smithi150.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T18:02:28.785 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T18:02:28.785 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T18:02:28.785 INFO:teuthology.orchestra.run.smithi150.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T18:02:28.785 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T18:02:28.785 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T18:02:28.786 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T18:02:28.786 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T18:02:28.786 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T18:02:28.786 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T18:02:28.786 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T18:02:28.786 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T18:02:28.786 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:02:28.787 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:02:28.787 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:02:28.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T18:02:28.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T18:02:28.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T18:02:28.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T18:02:28.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T18:02:28.787 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T18:02:28.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T18:02:28.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T18:02:28.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T18:02:28.788 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T18:02:28.789 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T18:02:28.789 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T18:02:28.789 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T18:02:28.789 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T18:02:28.789 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T18:02:28.789 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T18:02:28.789 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T18:02:28.789 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T18:02:28.790 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T18:02:28.790 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T18:02:28.790 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T18:02:28.790 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T18:02:28.790 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T18:02:28.790 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T18:02:28.791 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T18:02:28.791 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T18:02:28.791 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T18:02:28.791 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T18:02:28.791 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T18:02:28.791 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T18:02:28.791 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T18:02:28.792 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T18:02:28.792 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T18:02:28.792 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T18:02:28.792 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.792 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.792 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T18:02:28.792 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T18:02:28.793 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.793 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T18:02:28.793 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T18:02:28.793 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T18:02:28.793 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T18:02:28.793 INFO:teuthology.orchestra.run.smithi150.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T18:02:28.793 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T18:02:28.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T18:02:28.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T18:02:28.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T18:02:28.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T18:02:28.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T18:02:28.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T18:02:28.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T18:02:28.795 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.795 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.795 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:02:28.795 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T18:02:28.795 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T18:02:28.795 INFO:teuthology.orchestra.run.smithi150.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T18:02:28.795 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T18:02:28.795 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T18:02:28.796 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T18:02:28.796 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-07-14T18:02:28.796 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-07-14T18:02:29.857 DEBUG:teuthology.parallel:result is None 2023-07-14T18:02:29.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:02:35.219 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T18:02:35.248 INFO:teuthology.orchestra.run.smithi115.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T18:02:35.249 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T18:02:35.249 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T18:02:35.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:02:40.547 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T18:02:40.574 INFO:teuthology.orchestra.run.smithi150.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T18:02:40.575 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T18:02:40.575 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T18:02:40.576 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:02:45.985 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T18:02:46.015 INFO:teuthology.orchestra.run.smithi187.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T18:02:46.015 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T18:02:46.016 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T18:02:46.016 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-14T18:02:46.017 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:02:46.017 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T18:02:46.064 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T18:02:46.065 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T18:02:46.114 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T18:02:46.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T18:02:46.161 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-14T18:02:46.162 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:02:46.162 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T18:02:46.194 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T18:02:46.264 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T18:02:46.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T18:02:46.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T18:02:46.367 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T18:02:46.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T18:02:46.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T18:02:46.469 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-14T18:02:46.469 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:02:46.470 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T18:02:46.500 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T18:02:46.570 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T18:02:46.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T18:02:46.601 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T18:02:46.670 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T18:02:46.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T18:02:46.701 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T18:02:46.771 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-14T18:02:46.771 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:02:46.771 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T18:02:46.803 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T18:02:46.873 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T18:02:46.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T18:02:46.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T18:02:46.974 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T18:02:46.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T18:02:47.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T18:02:47.072 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-14T18:02:47.175 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-07-14T18:02:47.176 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:02:47.186 INFO:tasks.cephadm:Cluster fsid is a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:02:47.187 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-14T18:02:47.188 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.115', 'mon.b': '172.21.15.150', 'mon.c': '172.21.15.187'} 2023-07-14T18:02:47.188 INFO:tasks.cephadm:First mon is mon.a on smithi115 2023-07-14T18:02:47.188 INFO:tasks.cephadm:First mgr is x 2023-07-14T18:02:47.189 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-14T18:02:47.189 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2023-07-14T18:02:47.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2023-07-14T18:02:47.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo hostname $(hostname -s) 2023-07-14T18:02:47.281 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-14T18:02:47.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:02:50.820 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71431/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-4940-g81c9387d', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-4940.g81c9387d'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-14 16:50:26.249583', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230714.134335', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2023-07-14T18:02:51.105 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20230714.134335, sha1 81c9387d2e7576da065747ec9c16d1c7d69016ec from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:02:51.106 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm 2023-07-14T18:02:51.106 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T18:02:52.515 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 18:02 /home/ubuntu/cephtest/cephadm 2023-07-14T18:02:52.516 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T18:02:54.408 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 18:02 /home/ubuntu/cephtest/cephadm 2023-07-14T18:02:54.408 DEBUG:teuthology.orchestra.run.smithi187:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T18:02:56.140 INFO:teuthology.orchestra.run.smithi187.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 18:02 /home/ubuntu/cephtest/cephadm 2023-07-14T18:02:56.141 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T18:02:56.163 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T18:02:56.185 DEBUG:teuthology.orchestra.run.smithi187:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T18:02:56.219 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec on all hosts... 2023-07-14T18:02:56.219 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T18:02:56.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T18:02:56.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T18:02:57.474 INFO:teuthology.orchestra.run.smithi187.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:02:57.489 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:02:57.565 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:03:23.162 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:03:23.163 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:03:23.163 INFO:teuthology.orchestra.run.smithi115.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T18:03:23.163 INFO:teuthology.orchestra.run.smithi115.stdout: "repo_digests": [ 2023-07-14T18:03:23.163 INFO:teuthology.orchestra.run.smithi115.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T18:03:23.163 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2023-07-14T18:03:23.164 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:03:27.057 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-07-14T18:03:27.057 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:03:27.057 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T18:03:27.058 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2023-07-14T18:03:27.058 INFO:teuthology.orchestra.run.smithi150.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T18:03:27.058 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-07-14T18:03:27.058 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-07-14T18:03:30.386 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-07-14T18:03:30.386 INFO:teuthology.orchestra.run.smithi187.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:03:30.386 INFO:teuthology.orchestra.run.smithi187.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T18:03:30.387 INFO:teuthology.orchestra.run.smithi187.stdout: "repo_digests": [ 2023-07-14T18:03:30.387 INFO:teuthology.orchestra.run.smithi187.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T18:03:30.387 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-07-14T18:03:30.387 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-07-14T18:03:30.406 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2023-07-14T18:03:30.442 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2023-07-14T18:03:30.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph 2023-07-14T18:03:30.508 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2023-07-14T18:03:30.539 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2023-07-14T18:03:30.570 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /etc/ceph 2023-07-14T18:03:30.596 INFO:tasks.cephadm:Writing seed config... 2023-07-14T18:03:30.598 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-14T18:03:30.598 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-14T18:03:30.598 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-14T18:03:30.598 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-14T18:03:30.599 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-14T18:03:30.599 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-14T18:03:30.599 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-14T18:03:30.600 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-14T18:03:30.600 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-14T18:03:30.600 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-14T18:03:30.600 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-14T18:03:30.600 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-14T18:03:30.601 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-14T18:03:30.601 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-14T18:03:30.601 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-14T18:03:30.601 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-14T18:03:30.601 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-14T18:03:30.602 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-14T18:03:30.602 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-14T18:03:30.602 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-14T18:03:30.602 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-14T18:03:30.602 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-14T18:03:30.603 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-14T18:03:30.603 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-14T18:03:30.603 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-14T18:03:30.603 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-14T18:03:30.603 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-14T18:03:30.604 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-14T18:03:30.604 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-14T18:03:30.604 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-14T18:03:30.604 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-14T18:03:30.605 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-14T18:03:30.605 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-14T18:03:30.605 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-14T18:03:30.605 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-14T18:03:30.605 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-14T18:03:30.606 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-14T18:03:30.606 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-14T18:03:30.606 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-14T18:03:30.606 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-14T18:03:30.606 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-07-14T18:03:30.607 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:03:30.607 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-14T18:03:30.625 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a39f385c-2270-11ee-9b33-001a4aab830c [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-14T18:03:30.626 DEBUG:teuthology.orchestra.run.smithi115:mon.a> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.a.service 2023-07-14T18:03:30.668 DEBUG:teuthology.orchestra.run.smithi115:mgr.x> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.x.service 2023-07-14T18:03:30.696 INFO:journalctl@ceph.mon.a.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:03:30.697 INFO:tasks.cephadm:Bootstrapping... 2023-07-14T18:03:30.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec -v bootstrap --fsid a39f385c-2270-11ee-9b33-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.115 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:03:30.724 INFO:journalctl@ceph.mgr.x.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:03:30.858 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:03:30.858 INFO:teuthology.orchestra.run.smithi115.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec', '-v', 'bootstrap', '--fsid', 'a39f385c-2270-11ee-9b33-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.115', '--skip-admin-label'] 2023-07-14T18:03:31.008 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stdout 4.1.1 2023-07-14T18:03:31.008 INFO:teuthology.orchestra.run.smithi115.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-14T18:03:31.009 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying podman|docker is present... 2023-07-14T18:03:31.154 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stdout 4.1.1 2023-07-14T18:03:31.154 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying lvm2 is present... 2023-07-14T18:03:31.154 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying time synchronization is in place... 2023-07-14T18:03:31.161 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T18:03:31.162 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T18:03:31.167 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T18:03:31.167 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-07-14T18:03:31.172 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout enabled 2023-07-14T18:03:31.178 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout active 2023-07-14T18:03:31.178 INFO:teuthology.orchestra.run.smithi115.stdout:Unit chronyd.service is enabled and running 2023-07-14T18:03:31.178 INFO:teuthology.orchestra.run.smithi115.stdout:Repeating the final host check... 2023-07-14T18:03:31.318 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stdout 4.1.1 2023-07-14T18:03:31.318 INFO:teuthology.orchestra.run.smithi115.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-14T18:03:31.318 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl is present 2023-07-14T18:03:31.318 INFO:teuthology.orchestra.run.smithi115.stdout:lvcreate is present 2023-07-14T18:03:31.324 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T18:03:31.324 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T18:03:31.330 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T18:03:31.330 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-07-14T18:03:31.335 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout enabled 2023-07-14T18:03:31.341 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout active 2023-07-14T18:03:31.341 INFO:teuthology.orchestra.run.smithi115.stdout:Unit chronyd.service is enabled and running 2023-07-14T18:03:31.342 INFO:teuthology.orchestra.run.smithi115.stdout:Host looks OK 2023-07-14T18:03:31.342 INFO:teuthology.orchestra.run.smithi115.stdout:Cluster fsid: a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:03:31.342 INFO:teuthology.orchestra.run.smithi115.stdout:Acquiring lock 139797397305832 on /run/cephadm/a39f385c-2270-11ee-9b33-001a4aab830c.lock 2023-07-14T18:03:31.342 INFO:teuthology.orchestra.run.smithi115.stdout:Lock 139797397305832 acquired on /run/cephadm/a39f385c-2270-11ee-9b33-001a4aab830c.lock 2023-07-14T18:03:31.342 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying IP 172.21.15.115 port 3300 ... 2023-07-14T18:03:31.343 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying IP 172.21.15.115 port 6789 ... 2023-07-14T18:03:31.343 INFO:teuthology.orchestra.run.smithi115.stdout:Base mon IP(s) is [172.21.15.115:3300, 172.21.15.115:6789], mon addrv is [v2:172.21.15.115:3300,v1:172.21.15.115:6789] 2023-07-14T18:03:31.346 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-07-14T18:03:31.346 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.115 metric 100 2023-07-14T18:03:31.348 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-14T18:03:31.349 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-07-14T18:03:31.349 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-07-14T18:03:31.351 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-14T18:03:31.351 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-14T18:03:31.351 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T18:03:31.351 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-07-14T18:03:31.351 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout inet6 fe80::1d33:d16a:ed86:b5bf/64 scope link noprefixroute 2023-07-14T18:03:31.352 INFO:teuthology.orchestra.run.smithi115.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T18:03:31.352 INFO:teuthology.orchestra.run.smithi115.stdout:Mon IP `172.21.15.115` is in CIDR network `172.21.0.0/20` 2023-07-14T18:03:31.352 INFO:teuthology.orchestra.run.smithi115.stdout:Mon IP `172.21.15.115` is in CIDR network `172.21.0.0/20` 2023-07-14T18:03:31.352 INFO:teuthology.orchestra.run.smithi115.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-14T18:03:31.353 INFO:teuthology.orchestra.run.smithi115.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-14T18:03:31.354 INFO:teuthology.orchestra.run.smithi115.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:03:31.799 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stdout 219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T18:03:31.800 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:03:31.800 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Getting image source signatures 2023-07-14T18:03:31.800 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Copying blob sha256:0e454b422ba90e2ad13e636c40610401086a3576fe973b0460943839fe5f2508 2023-07-14T18:03:31.800 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-14T18:03:31.800 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Copying config sha256:219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T18:03:31.801 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-14T18:03:31.801 INFO:teuthology.orchestra.run.smithi115.stdout:/bin/podman: stderr Storing signatures 2023-07-14T18:03:32.555 INFO:teuthology.orchestra.run.smithi115.stdout:ceph: stdout ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T18:03:32.555 INFO:teuthology.orchestra.run.smithi115.stdout:Ceph version: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T18:03:32.555 INFO:teuthology.orchestra.run.smithi115.stdout:Extracting ceph user uid/gid from container image... 2023-07-14T18:03:33.305 INFO:teuthology.orchestra.run.smithi115.stdout:stat: stdout 167 167 2023-07-14T18:03:33.305 INFO:teuthology.orchestra.run.smithi115.stdout:Creating initial keys... 2023-07-14T18:03:34.063 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph-authtool: stdout AQD1jbFkln7EJRAATJFzayR9yniRdvJdUYeyWQ== 2023-07-14T18:03:34.714 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph-authtool: stdout AQD2jbFkaBlcFxAA1uGE2dVx0epkVbNOC9BIHQ== 2023-07-14T18:03:35.468 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph-authtool: stdout AQD3jbFk3rr8AxAAVyIYLq+MkqDbBWrXdQcufg== 2023-07-14T18:03:35.469 INFO:teuthology.orchestra.run.smithi115.stdout:Creating initial monmap... 2023-07-14T18:03:36.217 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T18:03:36.218 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-14T18:03:36.218 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:03:36.218 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T18:03:36.218 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool for a [v2:172.21.15.115:3300,v1:172.21.15.115:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T18:03:36.218 INFO:teuthology.orchestra.run.smithi115.stdout:setting min_mon_release = pacific 2023-07-14T18:03:36.218 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: set fsid to a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:03:36.218 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T18:03:36.218 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:03:36.219 INFO:teuthology.orchestra.run.smithi115.stdout:Creating mon... 2023-07-14T18:03:36.967 INFO:teuthology.orchestra.run.smithi115.stdout:create mon.a on 2023-07-14T18:03:37.094 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-14T18:03:37.180 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-14T18:03:37.297 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a39f385c-2270-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-a39f385c-2270-11ee-9b33-001a4aab830c.target. 2023-07-14T18:03:37.297 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a39f385c-2270-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-a39f385c-2270-11ee-9b33-001a4aab830c.target. 2023-07-14T18:03:37.464 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.a 2023-07-14T18:03:37.464 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.a.service: Unit ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.a.service not loaded. 2023-07-14T18:03:37.546 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a39f385c-2270-11ee-9b33-001a4aab830c.target.wants/ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-a39f385c-2270-11ee-9b33-001a4aab830c@.service. 2023-07-14T18:03:37.844 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:37 smithi115 systemd[1]: Starting Ceph mon.a for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:03:38.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:37 smithi115 podman[126912]: 2023-07-14T18:03:38.137 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:38 smithi115 bash[126912]: 78664e853e1d80ebe4c94a8b85ff28c304162bf872a560f21946d1cac5fb7ff5 2023-07-14T18:03:38.138 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:38 smithi115 ceph-mon[126952]: mkfs a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:03:38.138 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:38 smithi115 ceph-mon[126952]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T18:03:38.149 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:03:38.149 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout disabled 2023-07-14T18:03:38.156 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:03:38.156 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-07-14T18:03:38.156 INFO:teuthology.orchestra.run.smithi115.stdout:firewalld.service is not enabled 2023-07-14T18:03:38.156 INFO:teuthology.orchestra.run.smithi115.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T18:03:38.157 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mon to start... 2023-07-14T18:03:38.157 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mon... 2023-07-14T18:03:38.401 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:38 smithi115 systemd[1]: Started Ceph mon.a for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:03:39.106 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout cluster: 2023-07-14T18:03:39.106 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout id: a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:03:39.106 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-14T18:03:39.107 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:39.107 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout services: 2023-07-14T18:03:39.107 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.568401s) 2023-07-14T18:03:39.107 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-14T18:03:39.107 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-14T18:03:39.107 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:39.107 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout data: 2023-07-14T18:03:39.108 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-14T18:03:39.108 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-14T18:03:39.108 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-14T18:03:39.108 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout pgs: 2023-07-14T18:03:39.108 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:39.108 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.592+0000 7fd048edc700 1 Processor -- start 2023-07-14T18:03:39.108 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.592+0000 7fd048edc700 1 -- start start 2023-07-14T18:03:39.109 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.592+0000 7fd048edc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044105a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:39.109 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.592+0000 7fd048edc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd044105fc0 con 0x7fd044105670 2023-07-14T18:03:39.109 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.593+0000 7fd04259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044105a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:39.109 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.593+0000 7fd04259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044105a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51214/0 (socket says 172.21.15.115:51214) 2023-07-14T18:03:39.109 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.593+0000 7fd04259c700 1 -- 172.21.15.115:0/3702368415 learned_addr learned my addr 172.21.15.115:0/3702368415 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:39.109 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.593+0000 7fd04259c700 1 -- 172.21.15.115:0/3702368415 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd044106100 con 0x7fd044105670 2023-07-14T18:03:39.109 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.593+0000 7fd04259c700 1 --2- 172.21.15.115:0/3702368415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044105a80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd0340098a0 tx=0x7fd034009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff925e71c6a77b3e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:39.109 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.593+0000 7fd04159a700 1 -- 172.21.15.115:0/3702368415 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd034004160 con 0x7fd044105670 2023-07-14T18:03:39.110 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.593+0000 7fd04159a700 1 -- 172.21.15.115:0/3702368415 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0340042c0 con 0x7fd044105670 2023-07-14T18:03:39.110 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.593+0000 7fd04159a700 1 -- 172.21.15.115:0/3702368415 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0340045a0 con 0x7fd044105670 2023-07-14T18:03:39.110 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.594+0000 7fd048edc700 1 -- 172.21.15.115:0/3702368415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 msgr2=0x7fd044105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:39.110 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.594+0000 7fd048edc700 1 --2- 172.21.15.115:0/3702368415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044105a80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd0340098a0 tx=0x7fd034009bb0 comp rx=0 tx=0).stop 2023-07-14T18:03:39.110 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.594+0000 7fd048edc700 1 -- 172.21.15.115:0/3702368415 shutdown_connections 2023-07-14T18:03:39.110 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.594+0000 7fd048edc700 1 --2- 172.21.15.115:0/3702368415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044105a80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:39.110 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.594+0000 7fd048edc700 1 -- 172.21.15.115:0/3702368415 >> 172.21.15.115:0/3702368415 conn(0x7fd0440fee80 msgr2=0x7fd044101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:39.110 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.594+0000 7fd048edc700 1 -- 172.21.15.115:0/3702368415 shutdown_connections 2023-07-14T18:03:39.111 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.594+0000 7fd048edc700 1 -- 172.21.15.115:0/3702368415 wait complete. 2023-07-14T18:03:39.111 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.594+0000 7fd048edc700 1 Processor -- start 2023-07-14T18:03:39.111 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd048edc700 1 -- start start 2023-07-14T18:03:39.111 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd048edc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:39.111 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd048edc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd044198b50 con 0x7fd044105670 2023-07-14T18:03:39.111 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd04259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:39.111 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd04259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51216/0 (socket says 172.21.15.115:51216) 2023-07-14T18:03:39.112 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd04259c700 1 -- 172.21.15.115:0/3938428163 learned_addr learned my addr 172.21.15.115:0/3938428163 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:39.112 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd04259c700 1 -- 172.21.15.115:0/3938428163 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd034009550 con 0x7fd044105670 2023-07-14T18:03:39.112 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd04259c700 1 --2- 172.21.15.115:0/3938428163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044198610 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd0440661d0 tx=0x7fd034004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:39.112 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd03b7fe700 1 -- 172.21.15.115:0/3938428163 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd034011630 con 0x7fd044105670 2023-07-14T18:03:39.112 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd03b7fe700 1 -- 172.21.15.115:0/3938428163 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd034011790 con 0x7fd044105670 2023-07-14T18:03:39.112 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd03b7fe700 1 -- 172.21.15.115:0/3938428163 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd034011a30 con 0x7fd044105670 2023-07-14T18:03:39.112 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd048edc700 1 -- 172.21.15.115:0/3938428163 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd044198d50 con 0x7fd044105670 2023-07-14T18:03:39.112 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.595+0000 7fd048edc700 1 -- 172.21.15.115:0/3938428163 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd044199170 con 0x7fd044105670 2023-07-14T18:03:39.113 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.596+0000 7fd03b7fe700 1 -- 172.21.15.115:0/3938428163 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd034022070 con 0x7fd044105670 2023-07-14T18:03:39.113 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.596+0000 7fd03b7fe700 1 -- 172.21.15.115:0/3938428163 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd03401c450 con 0x7fd044105670 2023-07-14T18:03:39.113 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.596+0000 7fd048edc700 1 -- 172.21.15.115:0/3938428163 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd044191ed0 con 0x7fd044105670 2023-07-14T18:03:39.113 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.599+0000 7fd03b7fe700 1 -- 172.21.15.115:0/3938428163 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd03401b450 con 0x7fd044105670 2023-07-14T18:03:39.113 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.667+0000 7fd048edc700 1 -- 172.21.15.115:0/3938428163 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fd04402c8e0 con 0x7fd044105670 2023-07-14T18:03:39.113 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.667+0000 7fd03b7fe700 1 -- 172.21.15.115:0/3938428163 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fd03401b5f0 con 0x7fd044105670 2023-07-14T18:03:39.113 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.668+0000 7fd048edc700 1 -- 172.21.15.115:0/3938428163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 msgr2=0x7fd044198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:39.114 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.668+0000 7fd048edc700 1 --2- 172.21.15.115:0/3938428163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044198610 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd0440661d0 tx=0x7fd034004750 comp rx=0 tx=0).stop 2023-07-14T18:03:39.114 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.668+0000 7fd048edc700 1 -- 172.21.15.115:0/3938428163 shutdown_connections 2023-07-14T18:03:39.114 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.668+0000 7fd048edc700 1 --2- 172.21.15.115:0/3938428163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd044105670 0x7fd044198610 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:39.114 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.668+0000 7fd048edc700 1 -- 172.21.15.115:0/3938428163 >> 172.21.15.115:0/3938428163 conn(0x7fd0440fee80 msgr2=0x7fd04418bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:39.114 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.668+0000 7fd048edc700 1 -- 172.21.15.115:0/3938428163 shutdown_connections 2023-07-14T18:03:39.114 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:38.668+0000 7fd048edc700 1 -- 172.21.15.115:0/3938428163 wait complete. 2023-07-14T18:03:39.114 INFO:teuthology.orchestra.run.smithi115.stdout:mon is available 2023-07-14T18:03:39.114 INFO:teuthology.orchestra.run.smithi115.stdout:Assimilating anything we can from ceph.conf... 2023-07-14T18:03:39.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:39 smithi115 ceph-mon[126952]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T18:03:39.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:39 smithi115 ceph-mon[126952]: monmap e1: 1 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} removed_ranks: {} 2023-07-14T18:03:39.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:39 smithi115 ceph-mon[126952]: fsmap 2023-07-14T18:03:39.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:39 smithi115 ceph-mon[126952]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T18:03:39.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:39 smithi115 ceph-mon[126952]: mgrmap e1: no daemons active 2023-07-14T18:03:39.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:39 smithi115 ceph-mon[126952]: from='client.? 172.21.15.115:0/3938428163' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-14T18:03:40.052 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:40.052 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T18:03:40.052 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout fsid = a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:03:40.053 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.115:3300,v1:172.21.15.115:6789] 2023-07-14T18:03:40.053 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T18:03:40.053 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T18:03:40.053 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T18:03:40.053 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T18:03:40.053 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:40.053 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T18:03:40.053 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T18:03:40.054 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:40.054 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T18:03:40.054 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T18:03:40.054 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T18:03:40.054 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T18:03:40.054 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T18:03:40.054 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T18:03:40.055 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.541+0000 7f0bafcff700 1 Processor -- start 2023-07-14T18:03:40.055 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.541+0000 7f0bafcff700 1 -- start start 2023-07-14T18:03:40.055 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.542+0000 7f0bafcff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba81080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:40.055 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.542+0000 7f0bafcff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba8079cf0 con 0x7f0ba8105cf0 2023-07-14T18:03:40.055 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.542+0000 7f0bada9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba81080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:40.055 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.542+0000 7f0bada9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba81080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51218/0 (socket says 172.21.15.115:51218) 2023-07-14T18:03:40.055 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.542+0000 7f0bada9b700 1 -- 172.21.15.115:0/916441860 learned_addr learned my addr 172.21.15.115:0/916441860 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:40.055 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.542+0000 7f0bada9b700 1 -- 172.21.15.115:0/916441860 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8079e30 con 0x7f0ba8105cf0 2023-07-14T18:03:40.056 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.542+0000 7f0bada9b700 1 --2- 172.21.15.115:0/916441860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba81080d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b980098a0 tx=0x7f0b98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c511efbba4df9955 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:40.056 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.542+0000 7f0baca99700 1 -- 172.21.15.115:0/916441860 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98004160 con 0x7f0ba8105cf0 2023-07-14T18:03:40.056 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.543+0000 7f0baca99700 1 -- 172.21.15.115:0/916441860 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b980042c0 con 0x7f0ba8105cf0 2023-07-14T18:03:40.056 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.543+0000 7f0baca99700 1 -- 172.21.15.115:0/916441860 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b980045a0 con 0x7f0ba8105cf0 2023-07-14T18:03:40.056 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.543+0000 7f0bafcff700 1 -- 172.21.15.115:0/916441860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 msgr2=0x7f0ba81080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:40.056 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.543+0000 7f0bafcff700 1 --2- 172.21.15.115:0/916441860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba81080d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b980098a0 tx=0x7f0b98009bb0 comp rx=0 tx=0).stop 2023-07-14T18:03:40.056 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.543+0000 7f0bafcff700 1 -- 172.21.15.115:0/916441860 shutdown_connections 2023-07-14T18:03:40.057 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.543+0000 7f0bafcff700 1 --2- 172.21.15.115:0/916441860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba81080d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:40.057 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.543+0000 7f0bafcff700 1 -- 172.21.15.115:0/916441860 >> 172.21.15.115:0/916441860 conn(0x7f0ba80ff710 msgr2=0x7f0ba8101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:40.057 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.543+0000 7f0bafcff700 1 -- 172.21.15.115:0/916441860 shutdown_connections 2023-07-14T18:03:40.057 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.543+0000 7f0bafcff700 1 -- 172.21.15.115:0/916441860 wait complete. 2023-07-14T18:03:40.057 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0bafcff700 1 Processor -- start 2023-07-14T18:03:40.057 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0bafcff700 1 -- start start 2023-07-14T18:03:40.057 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0bafcff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:40.057 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0bafcff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba8198b70 con 0x7f0ba8105cf0 2023-07-14T18:03:40.058 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0bada9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:40.058 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0bada9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba8198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51220/0 (socket says 172.21.15.115:51220) 2023-07-14T18:03:40.058 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0bada9b700 1 -- 172.21.15.115:0/2093817644 learned_addr learned my addr 172.21.15.115:0/2093817644 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:40.058 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0bada9b700 1 -- 172.21.15.115:0/2093817644 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98009550 con 0x7f0ba8105cf0 2023-07-14T18:03:40.058 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0bada9b700 1 --2- 172.21.15.115:0/2093817644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba8198630 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0b98004000 tx=0x7f0b98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:40.058 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0b9effd700 1 -- 172.21.15.115:0/2093817644 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98011630 con 0x7f0ba8105cf0 2023-07-14T18:03:40.058 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.544+0000 7f0b9effd700 1 -- 172.21.15.115:0/2093817644 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b98011790 con 0x7f0ba8105cf0 2023-07-14T18:03:40.058 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.545+0000 7f0bafcff700 1 -- 172.21.15.115:0/2093817644 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba8198d70 con 0x7f0ba8105cf0 2023-07-14T18:03:40.059 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.545+0000 7f0b9effd700 1 -- 172.21.15.115:0/2093817644 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b98011a30 con 0x7f0ba8105cf0 2023-07-14T18:03:40.059 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.545+0000 7f0bafcff700 1 -- 172.21.15.115:0/2093817644 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba8199190 con 0x7f0ba8105cf0 2023-07-14T18:03:40.059 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.545+0000 7f0b9effd700 1 -- 172.21.15.115:0/2093817644 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0b98022070 con 0x7f0ba8105cf0 2023-07-14T18:03:40.059 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.545+0000 7f0b9effd700 1 -- 172.21.15.115:0/2093817644 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0b9801c450 con 0x7f0ba8105cf0 2023-07-14T18:03:40.059 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.545+0000 7f0bafcff700 1 -- 172.21.15.115:0/2093817644 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba8192460 con 0x7f0ba8105cf0 2023-07-14T18:03:40.059 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.548+0000 7f0b9effd700 1 -- 172.21.15.115:0/2093817644 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f0b9801b450 con 0x7f0ba8105cf0 2023-07-14T18:03:40.059 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.614+0000 7f0bafcff700 1 -- 172.21.15.115:0/2093817644 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0ba802cb60 con 0x7f0ba8105cf0 2023-07-14T18:03:40.060 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.618+0000 7f0b9effd700 1 -- 172.21.15.115:0/2093817644 <== mon.0 v2:172.21.15.115:3300/0 7 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f0b9801b5f0 con 0x7f0ba8105cf0 2023-07-14T18:03:40.060 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.618+0000 7f0b9effd700 1 -- 172.21.15.115:0/2093817644 <== mon.0 v2:172.21.15.115:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f0b980244d0 con 0x7f0ba8105cf0 2023-07-14T18:03:40.060 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.619+0000 7f0bafcff700 1 -- 172.21.15.115:0/2093817644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 msgr2=0x7f0ba8198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:40.060 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.619+0000 7f0bafcff700 1 --2- 172.21.15.115:0/2093817644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba8198630 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0b98004000 tx=0x7f0b98004750 comp rx=0 tx=0).stop 2023-07-14T18:03:40.060 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.619+0000 7f0bafcff700 1 -- 172.21.15.115:0/2093817644 shutdown_connections 2023-07-14T18:03:40.060 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.619+0000 7f0bafcff700 1 --2- 172.21.15.115:0/2093817644 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ba8105cf0 0x7f0ba8198630 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:40.060 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.619+0000 7f0bafcff700 1 -- 172.21.15.115:0/2093817644 >> 172.21.15.115:0/2093817644 conn(0x7f0ba80ff710 msgr2=0x7f0ba8100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:40.060 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.619+0000 7f0bafcff700 1 -- 172.21.15.115:0/2093817644 shutdown_connections 2023-07-14T18:03:40.061 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:39.619+0000 7f0bafcff700 1 -- 172.21.15.115:0/2093817644 wait complete. 2023-07-14T18:03:40.061 INFO:teuthology.orchestra.run.smithi115.stdout:Generating new minimal ceph.conf... 2023-07-14T18:03:40.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:40 smithi115 ceph-mon[126952]: from='client.? 172.21.15.115:0/2093817644' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T18:03:40.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:40 smithi115 ceph-mon[126952]: from='client.? 172.21.15.115:0/2093817644' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-14T18:03:40.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.529+0000 7f5cfc356700 1 Processor -- start 2023-07-14T18:03:40.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.529+0000 7f5cfc356700 1 -- start start 2023-07-14T18:03:40.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.529+0000 7f5cfc356700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf41080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:40.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.529+0000 7f5cfc356700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf4079d10 con 0x7f5cf4105cd0 2023-07-14T18:03:40.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.529+0000 7f5cfa0f2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf41080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:40.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.529+0000 7f5cfa0f2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf41080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51222/0 (socket says 172.21.15.115:51222) 2023-07-14T18:03:40.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.529+0000 7f5cfa0f2700 1 -- 172.21.15.115:0/873889705 learned_addr learned my addr 172.21.15.115:0/873889705 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:40.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.530+0000 7f5cfa0f2700 1 -- 172.21.15.115:0/873889705 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf4079e50 con 0x7f5cf4105cd0 2023-07-14T18:03:40.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.530+0000 7f5cfa0f2700 1 --2- 172.21.15.115:0/873889705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf41080b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00098a0 tx=0x7f5cf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8081f9cba88f249 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:40.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.530+0000 7f5cf90f0700 1 -- 172.21.15.115:0/873889705 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5cf0004160 con 0x7f5cf4105cd0 2023-07-14T18:03:40.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.530+0000 7f5cf90f0700 1 -- 172.21.15.115:0/873889705 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5cf00042c0 con 0x7f5cf4105cd0 2023-07-14T18:03:40.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.530+0000 7f5cf90f0700 1 -- 172.21.15.115:0/873889705 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5cf0011940 con 0x7f5cf4105cd0 2023-07-14T18:03:40.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.530+0000 7f5cfc356700 1 -- 172.21.15.115:0/873889705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 msgr2=0x7f5cf41080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:40.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.530+0000 7f5cfc356700 1 --2- 172.21.15.115:0/873889705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf41080b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00098a0 tx=0x7f5cf0009bb0 comp rx=0 tx=0).stop 2023-07-14T18:03:40.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfc356700 1 -- 172.21.15.115:0/873889705 shutdown_connections 2023-07-14T18:03:40.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfc356700 1 --2- 172.21.15.115:0/873889705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf41080b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:40.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfc356700 1 -- 172.21.15.115:0/873889705 >> 172.21.15.115:0/873889705 conn(0x7f5cf40ff710 msgr2=0x7f5cf4101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:40.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfc356700 1 -- 172.21.15.115:0/873889705 shutdown_connections 2023-07-14T18:03:40.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfc356700 1 -- 172.21.15.115:0/873889705 wait complete. 2023-07-14T18:03:40.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfc356700 1 Processor -- start 2023-07-14T18:03:40.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfc356700 1 -- start start 2023-07-14T18:03:40.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfc356700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf4198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:40.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfc356700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf4199230 con 0x7f5cf4105cd0 2023-07-14T18:03:40.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.531+0000 7f5cfa0f2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf4198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:40.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5cfa0f2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf4198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51224/0 (socket says 172.21.15.115:51224) 2023-07-14T18:03:41.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5cfa0f2700 1 -- 172.21.15.115:0/1284703959 learned_addr learned my addr 172.21.15.115:0/1284703959 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:41.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5cfa0f2700 1 -- 172.21.15.115:0/1284703959 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf0009550 con 0x7f5cf4105cd0 2023-07-14T18:03:41.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5cfa0f2700 1 --2- 172.21.15.115:0/1284703959 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf4198cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0000c00 tx=0x7f5cf0011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:41.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5ceb7fe700 1 -- 172.21.15.115:0/1284703959 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5cf00036a0 con 0x7f5cf4105cd0 2023-07-14T18:03:41.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5cfc356700 1 -- 172.21.15.115:0/1284703959 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf4199430 con 0x7f5cf4105cd0 2023-07-14T18:03:41.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5ceb7fe700 1 -- 172.21.15.115:0/1284703959 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5cf001b440 con 0x7f5cf4105cd0 2023-07-14T18:03:41.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5ceb7fe700 1 -- 172.21.15.115:0/1284703959 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5cf00193f0 con 0x7f5cf4105cd0 2023-07-14T18:03:41.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5cfc356700 1 -- 172.21.15.115:0/1284703959 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf41998b0 con 0x7f5cf4105cd0 2023-07-14T18:03:41.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5ceb7fe700 1 -- 172.21.15.115:0/1284703959 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5cf0019970 con 0x7f5cf4105cd0 2023-07-14T18:03:41.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.532+0000 7f5ceb7fe700 1 -- 172.21.15.115:0/1284703959 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5cf001a3d0 con 0x7f5cf4105cd0 2023-07-14T18:03:41.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.533+0000 7f5cfc356700 1 -- 172.21.15.115:0/1284703959 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf4061c70 con 0x7f5cf4105cd0 2023-07-14T18:03:41.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.535+0000 7f5ceb7fe700 1 -- 172.21.15.115:0/1284703959 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f5cf0029060 con 0x7f5cf4105cd0 2023-07-14T18:03:41.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.602+0000 7f5cfc356700 1 -- 172.21.15.115:0/1284703959 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5cf419c340 con 0x7f5cf4105cd0 2023-07-14T18:03:41.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.603+0000 7f5ceb7fe700 1 -- 172.21.15.115:0/1284703959 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f5cf001f070 con 0x7f5cf4105cd0 2023-07-14T18:03:41.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.603+0000 7f5cfc356700 1 -- 172.21.15.115:0/1284703959 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 msgr2=0x7f5cf4198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:41.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.603+0000 7f5cfc356700 1 --2- 172.21.15.115:0/1284703959 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf4198cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0000c00 tx=0x7f5cf0011e70 comp rx=0 tx=0).stop 2023-07-14T18:03:41.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.604+0000 7f5cfc356700 1 -- 172.21.15.115:0/1284703959 shutdown_connections 2023-07-14T18:03:41.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.604+0000 7f5cfc356700 1 --2- 172.21.15.115:0/1284703959 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5cf4105cd0 0x7f5cf4198cf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:41.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.604+0000 7f5cfc356700 1 -- 172.21.15.115:0/1284703959 >> 172.21.15.115:0/1284703959 conn(0x7f5cf40ff710 msgr2=0x7f5cf41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:41.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.604+0000 7f5cfc356700 1 -- 172.21.15.115:0/1284703959 shutdown_connections 2023-07-14T18:03:41.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:40.604+0000 7f5cfc356700 1 -- 172.21.15.115:0/1284703959 wait complete. 2023-07-14T18:03:41.002 INFO:teuthology.orchestra.run.smithi115.stdout:Restarting the monitor... 2023-07-14T18:03:41.329 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:40 smithi115 systemd[1]: Stopping Ceph mon.a for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:03:41.330 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:41 smithi115 ceph-mon[126952]: from='client.? 172.21.15.115:0/1284703959' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:03:41.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:41 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-a[126928]: 2023-07-14T18:03:41.301+0000 7f45a6b94700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:03:41.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:41 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-a[126928]: 2023-07-14T18:03:41.301+0000 7f45a6b94700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-14T18:03:41.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:41 smithi115 bash[127289]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-a 2023-07-14T18:03:42.116 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:41 smithi115 systemd[1]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.a.service: Succeeded. 2023-07-14T18:03:42.116 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:41 smithi115 systemd[1]: Stopped Ceph mon.a for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:03:42.116 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:41 smithi115 systemd[1]: Starting Ceph mon.a for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:03:42.342 INFO:teuthology.orchestra.run.smithi115.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-14T18:03:42.431 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 podman[127411]: 2023-07-14T18:03:42.432 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T18:03:42.432 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T18:03:42.432 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: pidfile_write: ignore empty --pid-file 2023-07-14T18:03:42.432 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 bash[127411]: 89625cb3df65dda21223654f107a7ec63e860c8ca56232b45f723393f0569681 2023-07-14T18:03:42.432 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: load: jerasure load: lrc 2023-07-14T18:03:42.433 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: RocksDB version: 7.9.2 2023-07-14T18:03:42.433 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Git sha 0 2023-07-14T18:03:42.433 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T18:03:42.433 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: DB SUMMARY 2023-07-14T18:03:42.433 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: DB Session ID: 8RKYI9T6V6FRRFG1EUNV 2023-07-14T18:03:42.433 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: CURRENT file: CURRENT 2023-07-14T18:03:42.433 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T18:03:42.433 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-14T18:03:42.434 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-14T18:03:42.434 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97184 ; 2023-07-14T18:03:42.434 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.error_if_exists: 0 2023-07-14T18:03:42.434 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.create_if_missing: 0 2023-07-14T18:03:42.434 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.paranoid_checks: 1 2023-07-14T18:03:42.434 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T18:03:42.434 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T18:03:42.435 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T18:03:42.435 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.env: 0x55a2e5e26c60 2023-07-14T18:03:42.435 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T18:03:42.435 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.info_log: 0x55a2e749d2a0 2023-07-14T18:03:42.435 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T18:03:42.435 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.statistics: (nil) 2023-07-14T18:03:42.435 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.use_fsync: 0 2023-07-14T18:03:42.435 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_log_file_size: 0 2023-07-14T18:03:42.436 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T18:03:42.436 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T18:03:42.436 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T18:03:42.436 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T18:03:42.436 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.allow_fallocate: 1 2023-07-14T18:03:42.436 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T18:03:42.437 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T18:03:42.437 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.use_direct_reads: 0 2023-07-14T18:03:42.437 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T18:03:42.437 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T18:03:42.437 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.db_log_dir: 2023-07-14T18:03:42.437 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.wal_dir: 2023-07-14T18:03:42.437 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T18:03:42.437 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.write_buffer_manager: 0x55a2e73be3c0 2023-07-14T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T18:03:42.439 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T18:03:42.439 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T18:03:42.439 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T18:03:42.439 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T18:03:42.439 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T18:03:42.439 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T18:03:42.439 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T18:03:42.439 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.unordered_write: 0 2023-07-14T18:03:42.440 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T18:03:42.440 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T18:03:42.440 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T18:03:42.440 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T18:03:42.440 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.row_cache: None 2023-07-14T18:03:42.440 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.wal_filter: None 2023-07-14T18:03:42.440 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T18:03:42.441 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T18:03:42.441 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.two_write_queues: 0 2023-07-14T18:03:42.441 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T18:03:42.441 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.wal_compression: 0 2023-07-14T18:03:42.441 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.atomic_flush: 0 2023-07-14T18:03:42.441 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T18:03:42.441 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T18:03:42.441 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T18:03:42.442 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.log_readahead_size: 0 2023-07-14T18:03:42.442 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T18:03:42.442 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T18:03:42.442 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T18:03:42.442 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T18:03:42.442 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T18:03:42.442 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T18:03:42.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T18:03:42.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_background_jobs: 2 2023-07-14T18:03:42.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_background_compactions: -1 2023-07-14T18:03:42.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_subcompactions: 1 2023-07-14T18:03:42.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T18:03:42.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T18:03:42.443 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T18:03:42.444 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T18:03:42.444 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T18:03:42.444 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T18:03:42.444 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T18:03:42.444 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T18:03:42.444 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_open_files: -1 2023-07-14T18:03:42.444 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T18:03:42.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T18:03:42.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T18:03:42.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T18:03:42.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_background_flushes: -1 2023-07-14T18:03:42.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Compression algorithms supported: 2023-07-14T18:03:42.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T18:03:42.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: kZSTD supported: 0 2023-07-14T18:03:42.445 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: kXpressCompression supported: 0 2023-07-14T18:03:42.446 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T18:03:42.446 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: kZlibCompression supported: 1 2023-07-14T18:03:42.446 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: kSnappyCompression supported: 1 2023-07-14T18:03:42.446 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: kLZ4Compression supported: 1 2023-07-14T18:03:42.446 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: kBZip2Compression supported: 0 2023-07-14T18:03:42.446 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T18:03:42.446 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T18:03:42.446 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-14T18:03:42.447 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T18:03:42.447 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T18:03:42.447 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.merge_operator: 2023-07-14T18:03:42.447 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_filter: None 2023-07-14T18:03:42.447 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T18:03:42.447 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T18:03:42.447 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T18:03:42.447 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T18:03:42.448 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a2e749d3a0) 2023-07-14T18:03:42.448 INFO:journalctl@ceph.mon.a.smithi115.stdout: cache_index_and_filter_blocks: 1 2023-07-14T18:03:42.448 INFO:journalctl@ceph.mon.a.smithi115.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T18:03:42.448 INFO:journalctl@ceph.mon.a.smithi115.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T18:03:42.448 INFO:journalctl@ceph.mon.a.smithi115.stdout: pin_top_level_index_and_filter: 1 2023-07-14T18:03:42.448 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_type: 0 2023-07-14T18:03:42.449 INFO:journalctl@ceph.mon.a.smithi115.stdout: data_block_index_type: 0 2023-07-14T18:03:42.449 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_shortening: 1 2023-07-14T18:03:42.449 INFO:journalctl@ceph.mon.a.smithi115.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T18:03:42.449 INFO:journalctl@ceph.mon.a.smithi115.stdout: checksum: 4 2023-07-14T18:03:42.449 INFO:journalctl@ceph.mon.a.smithi115.stdout: no_block_cache: 0 2023-07-14T18:03:42.449 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache: 0x55a2e745d1f0 2023-07-14T18:03:42.449 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_name: BinnedLRUCache 2023-07-14T18:03:42.450 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_options: 2023-07-14T18:03:42.450 INFO:journalctl@ceph.mon.a.smithi115.stdout: capacity : 536870912 2023-07-14T18:03:42.450 INFO:journalctl@ceph.mon.a.smithi115.stdout: num_shard_bits : 4 2023-07-14T18:03:42.450 INFO:journalctl@ceph.mon.a.smithi115.stdout: strict_capacity_limit : 0 2023-07-14T18:03:42.450 INFO:journalctl@ceph.mon.a.smithi115.stdout: high_pri_pool_ratio: 0.000 2023-07-14T18:03:42.450 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_compressed: (nil) 2023-07-14T18:03:42.450 INFO:journalctl@ceph.mon.a.smithi115.stdout: persistent_cache: (nil) 2023-07-14T18:03:42.450 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_size: 4096 2023-07-14T18:03:42.451 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_size_deviation: 10 2023-07-14T18:03:42.451 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_restart_interval: 16 2023-07-14T18:03:42.451 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_block_restart_interval: 1 2023-07-14T18:03:42.451 INFO:journalctl@ceph.mon.a.smithi115.stdout: metadata_block_size: 4096 2023-07-14T18:03:42.451 INFO:journalctl@ceph.mon.a.smithi115.stdout: partition_filters: 0 2023-07-14T18:03:42.451 INFO:journalctl@ceph.mon.a.smithi115.stdout: use_delta_encoding: 1 2023-07-14T18:03:42.451 INFO:journalctl@ceph.mon.a.smithi115.stdout: filter_policy: bloomfilter 2023-07-14T18:03:42.451 INFO:journalctl@ceph.mon.a.smithi115.stdout: whole_key_filtering: 1 2023-07-14T18:03:42.452 INFO:journalctl@ceph.mon.a.smithi115.stdout: verify_compression: 0 2023-07-14T18:03:42.452 INFO:journalctl@ceph.mon.a.smithi115.stdout: read_amp_bytes_per_bit: 0 2023-07-14T18:03:42.452 INFO:journalctl@ceph.mon.a.smithi115.stdout: format_version: 5 2023-07-14T18:03:42.452 INFO:journalctl@ceph.mon.a.smithi115.stdout: enable_index_compression: 1 2023-07-14T18:03:42.452 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_align: 0 2023-07-14T18:03:42.452 INFO:journalctl@ceph.mon.a.smithi115.stdout: max_auto_readahead_size: 262144 2023-07-14T18:03:42.452 INFO:journalctl@ceph.mon.a.smithi115.stdout: prepopulate_block_cache: 0 2023-07-14T18:03:42.452 INFO:journalctl@ceph.mon.a.smithi115.stdout: initial_auto_readahead_size: 8192 2023-07-14T18:03:42.453 INFO:journalctl@ceph.mon.a.smithi115.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T18:03:42.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T18:03:42.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T18:03:42.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression: NoCompression 2023-07-14T18:03:42.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T18:03:42.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T18:03:42.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T18:03:42.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.num_levels: 7 2023-07-14T18:03:42.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T18:03:42.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T18:03:42.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T18:03:42.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T18:03:42.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T18:03:42.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T18:03:42.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T18:03:42.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:03:42.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T18:03:42.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T18:03:42.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:03:42.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:03:42.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T18:03:42.455 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T18:03:42.456 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T18:03:42.456 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T18:03:42.456 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:03:42.456 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:03:42.456 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T18:03:42.456 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T18:03:42.456 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:03:42.456 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T18:03:42.457 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T18:03:42.457 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T18:03:42.457 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T18:03:42.457 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T18:03:42.457 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T18:03:42.457 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T18:03:42.457 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T18:03:42.457 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T18:03:42.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T18:03:42.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T18:03:42.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T18:03:42.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T18:03:42.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T18:03:42.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T18:03:42.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T18:03:42.458 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T18:03:42.459 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T18:03:42.459 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T18:03:42.459 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T18:03:42.459 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T18:03:42.459 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T18:03:42.459 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T18:03:42.459 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T18:03:42.459 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T18:03:42.460 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T18:03:42.460 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T18:03:42.460 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T18:03:42.460 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T18:03:42.460 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T18:03:42.460 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T18:03:42.460 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T18:03:42.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.table_properties_collectors: 2023-07-14T18:03:42.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.inplace_update_support: 0 2023-07-14T18:03:42.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T18:03:42.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T18:03:42.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T18:03:42.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T18:03:42.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.bloom_locality: 0 2023-07-14T18:03:42.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.max_successive_merges: 0 2023-07-14T18:03:42.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T18:03:42.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T18:03:42.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T18:03:42.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T18:03:42.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.ttl: 2592000 2023-07-14T18:03:42.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T18:03:42.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T18:03:42.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T18:03:42.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.enable_blob_files: false 2023-07-14T18:03:42.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.min_blob_size: 0 2023-07-14T18:03:42.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T18:03:42.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T18:03:42.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T18:03:42.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T18:03:42.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T18:03:42.463 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T18:03:42.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T18:03:42.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T18:03:42.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-14T18:03:42.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-14T18:03:42.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b7902fd7-8ac3-4cdb-96ce-99b3ac98d98f 2023-07-14T18:03:42.464 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689357822313666, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-14T18:03:42.465 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-14T18:03:42.465 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 systemd[1]: Started Ceph mon.a for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:03:42.465 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689357822315122, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92574, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 308, "table_properties": {"data_size": 90596, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13263, "raw_average_key_size": 49, "raw_value_size": 83349, "raw_average_value_size": 312, "num_data_blocks": 11, "num_entries": 267, "num_filter_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689357822, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b7902fd7-8ac3-4cdb-96ce-99b3ac98d98f", "db_session_id": "8RKYI9T6V6FRRFG1EUNV", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-14T18:03:42.465 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689357822315216, "job": 1, "event": "recovery_finished"} 2023-07-14T18:03:42.465 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-14T18:03:42.465 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T18:03:42.465 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a2e74f6000 2023-07-14T18:03:42.466 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: DB pointer 0x55a2e74e2000 2023-07-14T18:03:42.466 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T18:03:42.466 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T18:03:42.466 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** DB Stats ** 2023-07-14T18:03:42.466 INFO:journalctl@ceph.mon.a.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:03:42.466 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T18:03:42.466 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:03:42.466 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:03:42.467 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T18:03:42.467 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:03:42.467 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:03:42.467 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-07-14T18:03:42.467 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2023-07-14T18:03:42.467 INFO:journalctl@ceph.mon.a.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:03:42.467 INFO:journalctl@ceph.mon.a.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T18:03:42.468 INFO:journalctl@ceph.mon.a.smithi115.stdout: L0 2/0 92.24 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 99.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:03:42.468 INFO:journalctl@ceph.mon.a.smithi115.stdout: Sum 2/0 92.24 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 99.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:03:42.468 INFO:journalctl@ceph.mon.a.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 99.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:03:42.468 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-07-14T18:03:42.468 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2023-07-14T18:03:42.468 INFO:journalctl@ceph.mon.a.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:03:42.468 INFO:journalctl@ceph.mon.a.smithi115.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T18:03:42.469 INFO:journalctl@ceph.mon.a.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 99.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:03:42.469 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-07-14T18:03:42.469 INFO:journalctl@ceph.mon.a.smithi115.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T18:03:42.469 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-07-14T18:03:42.469 INFO:journalctl@ceph.mon.a.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:03:42.469 INFO:journalctl@ceph.mon.a.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T18:03:42.469 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T18:03:42.470 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T18:03:42.470 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T18:03:42.470 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T18:03:42.470 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative compaction: 0.00 GB write, 13.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:03:42.470 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval compaction: 0.00 GB write, 13.96 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:03:42.470 INFO:journalctl@ceph.mon.a.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T18:03:42.470 INFO:journalctl@ceph.mon.a.smithi115.stdout: Block cache BinnedLRUCache@0x55a2e745d1f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2023-07-14T18:03:42.470 INFO:journalctl@ceph.mon.a.smithi115.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-14T18:03:42.471 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-07-14T18:03:42.471 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T18:03:42.471 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: starting mon.a rank 0 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:03:42.471 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mon.a@-1(???) e1 preinit fsid a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:03:42.471 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mon.a@-1(???).mds e1 new map 2023-07-14T18:03:42.471 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mon.a@-1(???).mds e1 print_map 2023-07-14T18:03:42.471 INFO:journalctl@ceph.mon.a.smithi115.stdout: e1 2023-07-14T18:03:42.472 INFO:journalctl@ceph.mon.a.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:03:42.472 INFO:journalctl@ceph.mon.a.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:03:42.472 INFO:journalctl@ceph.mon.a.smithi115.stdout: legacy client fscid: -1 2023-07-14T18:03:42.472 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2023-07-14T18:03:42.472 INFO:journalctl@ceph.mon.a.smithi115.stdout: No filesystems configured 2023-07-14T18:03:42.472 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T18:03:42.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:03:42.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:03:42.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:03:42.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-14T18:03:42.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta expand map: {default=false} 2023-07-14T18:03:42.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta from 'false' to 'false' 2023-07-14T18:03:42.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta expanded map: {default=false} 2023-07-14T18:03:42.473 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta expand map: {default=info} 2023-07-14T18:03:42.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta from 'info' to 'info' 2023-07-14T18:03:42.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta expanded map: {default=info} 2023-07-14T18:03:42.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta expand map: {default=daemon} 2023-07-14T18:03:42.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T18:03:42.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T18:03:42.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta expand map: {default=debug} 2023-07-14T18:03:42.474 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T18:03:42.475 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: expand_channel_meta expanded map: {default=debug} 2023-07-14T18:03:42.475 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T18:03:42.475 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: monmap e1: 1 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} removed_ranks: {} 2023-07-14T18:03:42.475 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: fsmap 2023-07-14T18:03:42.475 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T18:03:42.475 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:42 smithi115 ceph-mon[127451]: mgrmap e1: no daemons active 2023-07-14T18:03:44.796 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.330+0000 7f9e8599e700 1 Processor -- start 2023-07-14T18:03:44.797 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.330+0000 7f9e8599e700 1 -- start start 2023-07-14T18:03:44.797 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.330+0000 7f9e8599e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80107d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:44.797 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.330+0000 7f9e8599e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e801082b0 con 0x7f9e80079c70 2023-07-14T18:03:44.797 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.331+0000 7f9e7effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80107d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:44.797 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.331+0000 7f9e7effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80107d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51226/0 (socket says 172.21.15.115:51226) 2023-07-14T18:03:44.797 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.331+0000 7f9e7effd700 1 -- 172.21.15.115:0/47991550 learned_addr learned my addr 172.21.15.115:0/47991550 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:44.797 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.331+0000 7f9e7effd700 1 -- 172.21.15.115:0/47991550 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e801083f0 con 0x7f9e80079c70 2023-07-14T18:03:44.798 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.331+0000 7f9e7effd700 1 --2- 172.21.15.115:0/47991550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80107d70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9e700098a0 tx=0x7f9e70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e98046e626d77c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:44.798 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.331+0000 7f9e7dffb700 1 -- 172.21.15.115:0/47991550 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e70004160 con 0x7f9e80079c70 2023-07-14T18:03:44.798 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.331+0000 7f9e7dffb700 1 -- 172.21.15.115:0/47991550 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f9e700042c0 con 0x7f9e80079c70 2023-07-14T18:03:44.798 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.331+0000 7f9e7dffb700 1 -- 172.21.15.115:0/47991550 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e70011940 con 0x7f9e80079c70 2023-07-14T18:03:44.798 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.332+0000 7f9e8599e700 1 -- 172.21.15.115:0/47991550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 msgr2=0x7f9e80107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:44.798 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.332+0000 7f9e8599e700 1 --2- 172.21.15.115:0/47991550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80107d70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9e700098a0 tx=0x7f9e70009bb0 comp rx=0 tx=0).stop 2023-07-14T18:03:44.798 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.332+0000 7f9e8599e700 1 -- 172.21.15.115:0/47991550 shutdown_connections 2023-07-14T18:03:44.798 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.332+0000 7f9e8599e700 1 --2- 172.21.15.115:0/47991550 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80107d70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:44.799 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.332+0000 7f9e8599e700 1 -- 172.21.15.115:0/47991550 >> 172.21.15.115:0/47991550 conn(0x7f9e800ff6e0 msgr2=0x7f9e80101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:44.799 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.332+0000 7f9e8599e700 1 -- 172.21.15.115:0/47991550 shutdown_connections 2023-07-14T18:03:44.799 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.332+0000 7f9e8599e700 1 -- 172.21.15.115:0/47991550 wait complete. 2023-07-14T18:03:44.799 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.332+0000 7f9e8599e700 1 Processor -- start 2023-07-14T18:03:44.799 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.333+0000 7f9e8599e700 1 -- start start 2023-07-14T18:03:44.799 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.333+0000 7f9e8599e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:44.799 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.333+0000 7f9e8599e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e80199220 con 0x7f9e80079c70 2023-07-14T18:03:44.799 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.333+0000 7f9e7effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:44.800 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.333+0000 7f9e7effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80198ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51228/0 (socket says 172.21.15.115:51228) 2023-07-14T18:03:44.800 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.333+0000 7f9e7effd700 1 -- 172.21.15.115:0/1814159244 learned_addr learned my addr 172.21.15.115:0/1814159244 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:44.800 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.333+0000 7f9e7effd700 1 -- 172.21.15.115:0/1814159244 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e70009550 con 0x7f9e80079c70 2023-07-14T18:03:44.800 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.333+0000 7f9e7effd700 1 --2- 172.21.15.115:0/1814159244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80198ce0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9e70000c00 tx=0x7f9e70011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:44.800 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.334+0000 7f9e8499c700 1 -- 172.21.15.115:0/1814159244 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e700036a0 con 0x7f9e80079c70 2023-07-14T18:03:44.800 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.334+0000 7f9e8499c700 1 -- 172.21.15.115:0/1814159244 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f9e7001b440 con 0x7f9e80079c70 2023-07-14T18:03:44.801 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.334+0000 7f9e8599e700 1 -- 172.21.15.115:0/1814159244 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e80199420 con 0x7f9e80079c70 2023-07-14T18:03:44.801 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.334+0000 7f9e8499c700 1 -- 172.21.15.115:0/1814159244 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e700193f0 con 0x7f9e80079c70 2023-07-14T18:03:44.801 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.334+0000 7f9e8599e700 1 -- 172.21.15.115:0/1814159244 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e80199840 con 0x7f9e80079c70 2023-07-14T18:03:44.801 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.334+0000 7f9e8499c700 1 -- 172.21.15.115:0/1814159244 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9e70019930 con 0x7f9e80079c70 2023-07-14T18:03:44.801 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.334+0000 7f9e8499c700 1 -- 172.21.15.115:0/1814159244 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9e7001a3d0 con 0x7f9e80079c70 2023-07-14T18:03:44.801 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.335+0000 7f9e8599e700 1 -- 172.21.15.115:0/1814159244 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e80061c70 con 0x7f9e80079c70 2023-07-14T18:03:44.801 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.337+0000 7f9e8499c700 1 -- 172.21.15.115:0/1814159244 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f9e70029060 con 0x7f9e80079c70 2023-07-14T18:03:44.801 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.403+0000 7f9e8599e700 1 -- 172.21.15.115:0/1814159244 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9e8019c240 con 0x7f9e80079c70 2023-07-14T18:03:44.802 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.405+0000 7f9e8499c700 1 -- 172.21.15.115:0/1814159244 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9e7001f070 con 0x7f9e80079c70 2023-07-14T18:03:44.802 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.406+0000 7f9e8599e700 1 -- 172.21.15.115:0/1814159244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 msgr2=0x7f9e80198ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:44.802 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.406+0000 7f9e8599e700 1 --2- 172.21.15.115:0/1814159244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80198ce0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9e70000c00 tx=0x7f9e70011e70 comp rx=0 tx=0).stop 2023-07-14T18:03:44.802 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.406+0000 7f9e8599e700 1 -- 172.21.15.115:0/1814159244 shutdown_connections 2023-07-14T18:03:44.802 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.406+0000 7f9e8599e700 1 --2- 172.21.15.115:0/1814159244 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9e80079c70 0x7f9e80198ce0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:44.802 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.406+0000 7f9e8599e700 1 -- 172.21.15.115:0/1814159244 >> 172.21.15.115:0/1814159244 conn(0x7f9e800ff6e0 msgr2=0x7f9e80100350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:44.802 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.406+0000 7f9e8599e700 1 -- 172.21.15.115:0/1814159244 shutdown_connections 2023-07-14T18:03:44.802 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:44.406+0000 7f9e8599e700 1 -- 172.21.15.115:0/1814159244 wait complete. 2023-07-14T18:03:44.803 INFO:teuthology.orchestra.run.smithi115.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-14T18:03:44.831 INFO:teuthology.orchestra.run.smithi115.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:03:44.831 INFO:teuthology.orchestra.run.smithi115.stdout:Creating mgr... 2023-07-14T18:03:44.831 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying port 9283 ... 2023-07-14T18:03:44.831 INFO:teuthology.orchestra.run.smithi115.stdout:Verifying port 8765 ... 2023-07-14T18:03:45.027 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T18:03:45.027 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.x.service: Unit ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.x.service not loaded. 2023-07-14T18:03:45.118 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a39f385c-2270-11ee-9b33-001a4aab830c.target.wants/ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-a39f385c-2270-11ee-9b33-001a4aab830c@.service. 2023-07-14T18:03:45.344 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:45 smithi115 systemd[1]: Starting Ceph mgr.x for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:03:45.692 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:45 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1814159244' entity='client.admin' 2023-07-14T18:03:45.693 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:45 smithi115 podman[127679]: 2023-07-14T18:03:45.723 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:03:45.723 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout disabled 2023-07-14T18:03:45.731 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:03:45.732 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-07-14T18:03:45.732 INFO:teuthology.orchestra.run.smithi115.stdout:firewalld.service is not enabled 2023-07-14T18:03:45.732 INFO:teuthology.orchestra.run.smithi115.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T18:03:45.741 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:03:45.741 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout disabled 2023-07-14T18:03:45.750 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:03:45.750 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-07-14T18:03:45.750 INFO:teuthology.orchestra.run.smithi115.stdout:firewalld.service is not enabled 2023-07-14T18:03:45.750 INFO:teuthology.orchestra.run.smithi115.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-14T18:03:45.750 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mgr to start... 2023-07-14T18:03:45.750 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mgr... 2023-07-14T18:03:46.002 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:45 smithi115 bash[127679]: 8b6d3c80ebc21eca04f1ad3698764b2a316664ab9601b143129444f84d3867e0 2023-07-14T18:03:46.002 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:45 smithi115 systemd[1]: Started Ceph mgr.x for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:03:46.003 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:45 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:45.805+0000 7f0e0c119200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:03:46.003 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:45 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:45.884+0000 7f0e0c119200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:03:46.003 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:45 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:45.974+0000 7f0e0c119200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:03:46.289 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:46 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:46.078+0000 7f0e0c119200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:03:46.560 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:46 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1619415652' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:03:46.560 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:46 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:46.261+0000 7f0e0c119200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:03:46.809 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:46.810 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:03:46.810 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:03:46.810 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:03:46.810 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:03:46.810 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:03:46.810 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:03:46.810 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:46.811 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:03:46.811 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:03:46.811 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:03:46.811 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:03:46.811 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:03:46.811 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:03:46.811 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:03:46.811 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2023-07-14T18:03:46.812 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:03:46.812 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:46.812 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:03:46.812 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:03:46.812 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:46.812 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:03:46.812 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:46.813 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:03:46.813 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:03:46.813 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:03:46.818 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:03:46.818 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:03:46.818 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:03:46.819 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:46.819 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:03:46.819 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:03:46.819 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:03:46.819 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:03:46.819 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:03:46.819 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:03:46.819 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:03:46.820 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:03:46.820 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:03:46.820 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:46.820 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:03:46.820 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:46.820 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:03:46.820 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:03:46.821 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:46.821 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:03:46.821 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T18:03:46.821 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:03:46.821 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:03:46.821 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:03:46.821 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:03:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:03:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:03:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:03:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:03:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:03:38.101692+0000", 2023-07-14T18:03:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:03:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:03:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:03:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.314+0000 7f6f606c4700 1 Processor -- start 2023-07-14T18:03:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.314+0000 7f6f606c4700 1 -- start start 2023-07-14T18:03:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.315+0000 7f6f606c4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f581080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.315+0000 7f6f606c4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58079d10 con 0x7f6f58105cd0 2023-07-14T18:03:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.315+0000 7f6f5e460700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f581080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.315+0000 7f6f5e460700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f581080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51234/0 (socket says 172.21.15.115:51234) 2023-07-14T18:03:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.315+0000 7f6f5e460700 1 -- 172.21.15.115:0/692356238 learned_addr learned my addr 172.21.15.115:0/692356238 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.315+0000 7f6f5e460700 1 -- 172.21.15.115:0/692356238 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f58079e50 con 0x7f6f58105cd0 2023-07-14T18:03:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.315+0000 7f6f5e460700 1 --2- 172.21.15.115:0/692356238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f581080b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6f540098a0 tx=0x7f6f54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1830fc640f507b5f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.316+0000 7f6f5d45e700 1 -- 172.21.15.115:0/692356238 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f54004160 con 0x7f6f58105cd0 2023-07-14T18:03:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.316+0000 7f6f5d45e700 1 -- 172.21.15.115:0/692356238 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6f540042c0 con 0x7f6f58105cd0 2023-07-14T18:03:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.316+0000 7f6f5d45e700 1 -- 172.21.15.115:0/692356238 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f54011940 con 0x7f6f58105cd0 2023-07-14T18:03:46.825 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.316+0000 7f6f606c4700 1 -- 172.21.15.115:0/692356238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 msgr2=0x7f6f581080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:46.825 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.316+0000 7f6f606c4700 1 --2- 172.21.15.115:0/692356238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f581080b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6f540098a0 tx=0x7f6f54009bb0 comp rx=0 tx=0).stop 2023-07-14T18:03:46.825 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.316+0000 7f6f606c4700 1 -- 172.21.15.115:0/692356238 shutdown_connections 2023-07-14T18:03:46.825 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.316+0000 7f6f606c4700 1 --2- 172.21.15.115:0/692356238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f581080b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:46.825 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.316+0000 7f6f606c4700 1 -- 172.21.15.115:0/692356238 >> 172.21.15.115:0/692356238 conn(0x7f6f580ff710 msgr2=0x7f6f58101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:46.825 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.316+0000 7f6f606c4700 1 -- 172.21.15.115:0/692356238 shutdown_connections 2023-07-14T18:03:46.825 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.317+0000 7f6f606c4700 1 -- 172.21.15.115:0/692356238 wait complete. 2023-07-14T18:03:46.825 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.317+0000 7f6f606c4700 1 Processor -- start 2023-07-14T18:03:46.826 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.317+0000 7f6f606c4700 1 -- start start 2023-07-14T18:03:46.826 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.317+0000 7f6f606c4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f58198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:46.826 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.317+0000 7f6f606c4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58199230 con 0x7f6f58105cd0 2023-07-14T18:03:46.826 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.317+0000 7f6f5e460700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f58198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:46.826 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.317+0000 7f6f5e460700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f58198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51236/0 (socket says 172.21.15.115:51236) 2023-07-14T18:03:46.826 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.317+0000 7f6f5e460700 1 -- 172.21.15.115:0/1619415652 learned_addr learned my addr 172.21.15.115:0/1619415652 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:46.826 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.318+0000 7f6f5e460700 1 -- 172.21.15.115:0/1619415652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f54009550 con 0x7f6f58105cd0 2023-07-14T18:03:46.827 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.318+0000 7f6f5e460700 1 --2- 172.21.15.115:0/1619415652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f58198cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f54006af0 tx=0x7f6f54011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:46.827 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.318+0000 7f6f4f7fe700 1 -- 172.21.15.115:0/1619415652 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f54003bf0 con 0x7f6f58105cd0 2023-07-14T18:03:46.827 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.318+0000 7f6f4f7fe700 1 -- 172.21.15.115:0/1619415652 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6f5401b8b0 con 0x7f6f58105cd0 2023-07-14T18:03:46.827 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.318+0000 7f6f606c4700 1 -- 172.21.15.115:0/1619415652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f58199490 con 0x7f6f58105cd0 2023-07-14T18:03:46.827 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.318+0000 7f6f4f7fe700 1 -- 172.21.15.115:0/1619415652 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f540198b0 con 0x7f6f58105cd0 2023-07-14T18:03:46.827 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.318+0000 7f6f606c4700 1 -- 172.21.15.115:0/1619415652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f58199910 con 0x7f6f58105cd0 2023-07-14T18:03:46.827 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.318+0000 7f6f4f7fe700 1 -- 172.21.15.115:0/1619415652 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6f540233f0 con 0x7f6f58105cd0 2023-07-14T18:03:46.827 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.318+0000 7f6f4f7fe700 1 -- 172.21.15.115:0/1619415652 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6f5401a710 con 0x7f6f58105cd0 2023-07-14T18:03:46.828 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.319+0000 7f6f606c4700 1 -- 172.21.15.115:0/1619415652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f58109190 con 0x7f6f58105cd0 2023-07-14T18:03:46.828 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.321+0000 7f6f4f7fe700 1 -- 172.21.15.115:0/1619415652 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f6f54028060 con 0x7f6f58105cd0 2023-07-14T18:03:46.828 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.393+0000 7f6f606c4700 1 -- 172.21.15.115:0/1619415652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6f5819c8e0 con 0x7f6f58105cd0 2023-07-14T18:03:46.828 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.394+0000 7f6f4f7fe700 1 -- 172.21.15.115:0/1619415652 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f6f5401bd70 con 0x7f6f58105cd0 2023-07-14T18:03:46.828 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.394+0000 7f6f606c4700 1 -- 172.21.15.115:0/1619415652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 msgr2=0x7f6f58198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:46.828 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.394+0000 7f6f606c4700 1 --2- 172.21.15.115:0/1619415652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f58198cf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f54006af0 tx=0x7f6f54011a30 comp rx=0 tx=0).stop 2023-07-14T18:03:46.829 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.395+0000 7f6f606c4700 1 -- 172.21.15.115:0/1619415652 shutdown_connections 2023-07-14T18:03:46.829 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.395+0000 7f6f606c4700 1 --2- 172.21.15.115:0/1619415652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f58105cd0 0x7f6f58198cf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:46.829 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.395+0000 7f6f606c4700 1 -- 172.21.15.115:0/1619415652 >> 172.21.15.115:0/1619415652 conn(0x7f6f580ff710 msgr2=0x7f6f581003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:46.830 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.395+0000 7f6f606c4700 1 -- 172.21.15.115:0/1619415652 shutdown_connections 2023-07-14T18:03:46.830 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:46 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:46.589+0000 7f0e0c119200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:03:46.830 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:46 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:46.679+0000 7f0e0c119200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:03:46.831 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:46.395+0000 7f6f606c4700 1 -- 172.21.15.115:0/1619415652 wait complete. 2023-07-14T18:03:46.831 INFO:teuthology.orchestra.run.smithi115.stdout:mgr not available, waiting (1/15)... 2023-07-14T18:03:47.094 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:46 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:46.808+0000 7f0e0c119200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:03:47.094 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:46 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:46.902+0000 7f0e0c119200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:03:47.844 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:47 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:47.318+0000 7f0e0c119200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:03:48.344 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:47 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:47.887+0000 7f0e0c119200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:03:48.344 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:48 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:48.055+0000 7f0e0c119200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:03:49.128 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:48 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:48.830+0000 7f0e0c119200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:03:49.129 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:48 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:48.920+0000 7f0e0c119200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:03:49.129 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:49 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:49.100+0000 7f0e0c119200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:03:49.802 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:49 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2165218182' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:03:49.803 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:49 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:49.387+0000 7f0e0c119200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:03:49.803 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:49 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:49.625+0000 7f0e0c119200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:03:49.803 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:49 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:49.774+0000 7f0e0c119200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:03:50.344 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:50 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:50.033+0000 7f0e0c119200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:03:51.072 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:50 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:50.726+0000 7f0e0c119200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:03:51.072 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:50 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:50.857+0000 7f0e0c119200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:03:51.344 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:51 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:51.044+0000 7f0e0c119200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:03:51.345 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:51 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:51.157+0000 7f0e0c119200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:03:51.564 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:51.565 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:03:51.565 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:03:51.565 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:03:51.565 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:03:51.565 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:03:51.566 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:03:51.566 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:51.566 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:03:51.566 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:03:51.566 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:03:51.566 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:03:51.566 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:03:51.567 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:03:51.567 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:03:51.567 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2023-07-14T18:03:51.567 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:03:51.567 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:51.567 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:03:51.568 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:03:51.568 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:51.568 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:03:51.568 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:51.568 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:03:51.568 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:03:51.568 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:03:51.574 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:03:51.574 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:03:51.575 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:03:51.575 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:51.575 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:03:51.575 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:03:51.575 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:03:51.575 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:03:51.576 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:03:51.576 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:03:51.576 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:03:51.576 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:03:51.576 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:03:51.576 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:51.576 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:03:51.577 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:51.577 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:03:51.577 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:03:51.577 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:51.577 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:03:51.577 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T18:03:51.578 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:03:51.578 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:03:51.578 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:03:51.578 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:03:51.578 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:03:51.578 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:03:51.578 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:03:51.578 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:51.579 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:03:51.579 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:51.579 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:03:38.101692+0000", 2023-07-14T18:03:51.579 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:03:51.579 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:51.579 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:03:51.579 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:03:51.580 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.320+0000 7f6dfc9ce700 1 Processor -- start 2023-07-14T18:03:51.580 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.320+0000 7f6dfc9ce700 1 -- start start 2023-07-14T18:03:51.580 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.320+0000 7f6dfc9ce700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df8108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:51.580 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.320+0000 7f6dfc9ce700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df8079d50 con 0x7f6df8105c90 2023-07-14T18:03:51.580 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.321+0000 7f6df659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df8108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:51.580 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.321+0000 7f6df659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df8108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51238/0 (socket says 172.21.15.115:51238) 2023-07-14T18:03:51.580 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.321+0000 7f6df659c700 1 -- 172.21.15.115:0/950659774 learned_addr learned my addr 172.21.15.115:0/950659774 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:51.581 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.321+0000 7f6df659c700 1 -- 172.21.15.115:0/950659774 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df8079e90 con 0x7f6df8105c90 2023-07-14T18:03:51.581 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.321+0000 7f6df659c700 1 --2- 172.21.15.115:0/950659774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df8108070 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6de80098a0 tx=0x7f6de8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e34057139fe4e2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:51.581 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.321+0000 7f6df559a700 1 -- 172.21.15.115:0/950659774 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6de8004160 con 0x7f6df8105c90 2023-07-14T18:03:51.581 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.321+0000 7f6df559a700 1 -- 172.21.15.115:0/950659774 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6de80042c0 con 0x7f6df8105c90 2023-07-14T18:03:51.581 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.321+0000 7f6df559a700 1 -- 172.21.15.115:0/950659774 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6de8011940 con 0x7f6df8105c90 2023-07-14T18:03:51.581 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.322+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/950659774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 msgr2=0x7f6df8108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:51.581 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.322+0000 7f6dfc9ce700 1 --2- 172.21.15.115:0/950659774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df8108070 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6de80098a0 tx=0x7f6de8009bb0 comp rx=0 tx=0).stop 2023-07-14T18:03:51.582 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.322+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/950659774 shutdown_connections 2023-07-14T18:03:51.582 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.322+0000 7f6dfc9ce700 1 --2- 172.21.15.115:0/950659774 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df8108070 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:51.582 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.322+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/950659774 >> 172.21.15.115:0/950659774 conn(0x7f6df80ff710 msgr2=0x7f6df8101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:51.582 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.322+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/950659774 shutdown_connections 2023-07-14T18:03:51.582 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.322+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/950659774 wait complete. 2023-07-14T18:03:51.582 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6dfc9ce700 1 Processor -- start 2023-07-14T18:03:51.582 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6dfc9ce700 1 -- start start 2023-07-14T18:03:51.583 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6dfc9ce700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df819af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:51.583 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6dfc9ce700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df819b460 con 0x7f6df8105c90 2023-07-14T18:03:51.583 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6df659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df819af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:51.583 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6df659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df819af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51240/0 (socket says 172.21.15.115:51240) 2023-07-14T18:03:51.583 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6df659c700 1 -- 172.21.15.115:0/2165218182 learned_addr learned my addr 172.21.15.115:0/2165218182 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:51.583 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6df659c700 1 -- 172.21.15.115:0/2165218182 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de8009550 con 0x7f6df8105c90 2023-07-14T18:03:51.583 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6df659c700 1 --2- 172.21.15.115:0/2165218182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df819af20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6de8000c00 tx=0x7f6de8011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:51.584 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6de77fe700 1 -- 172.21.15.115:0/2165218182 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6de80036a0 con 0x7f6df8105c90 2023-07-14T18:03:51.584 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/2165218182 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df819b660 con 0x7f6df8105c90 2023-07-14T18:03:51.584 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6de77fe700 1 -- 172.21.15.115:0/2165218182 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6de801b440 con 0x7f6df8105c90 2023-07-14T18:03:51.584 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.323+0000 7f6de77fe700 1 -- 172.21.15.115:0/2165218182 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6de80193f0 con 0x7f6df8105c90 2023-07-14T18:03:51.584 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.324+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/2165218182 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df819ba80 con 0x7f6df8105c90 2023-07-14T18:03:51.584 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.324+0000 7f6de77fe700 1 -- 172.21.15.115:0/2165218182 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6de8019930 con 0x7f6df8105c90 2023-07-14T18:03:51.584 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.324+0000 7f6de77fe700 1 -- 172.21.15.115:0/2165218182 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6de801a3d0 con 0x7f6df8105c90 2023-07-14T18:03:51.585 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.324+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/2165218182 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df8061c70 con 0x7f6df8105c90 2023-07-14T18:03:51.585 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.327+0000 7f6de77fe700 1 -- 172.21.15.115:0/2165218182 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f6de8029060 con 0x7f6df8105c90 2023-07-14T18:03:51.585 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.399+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/2165218182 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6df819e5b0 con 0x7f6df8105c90 2023-07-14T18:03:51.585 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.399+0000 7f6de77fe700 1 -- 172.21.15.115:0/2165218182 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f6de801f070 con 0x7f6df8105c90 2023-07-14T18:03:51.585 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.400+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/2165218182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 msgr2=0x7f6df819af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:51.585 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.400+0000 7f6dfc9ce700 1 --2- 172.21.15.115:0/2165218182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df819af20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6de8000c00 tx=0x7f6de8011e70 comp rx=0 tx=0).stop 2023-07-14T18:03:51.585 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.400+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/2165218182 shutdown_connections 2023-07-14T18:03:51.586 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.400+0000 7f6dfc9ce700 1 --2- 172.21.15.115:0/2165218182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6df8105c90 0x7f6df819af20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:51.586 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.400+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/2165218182 >> 172.21.15.115:0/2165218182 conn(0x7f6df80ff710 msgr2=0x7f6df81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:51.586 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.400+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/2165218182 shutdown_connections 2023-07-14T18:03:51.586 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:49.400+0000 7f6dfc9ce700 1 -- 172.21.15.115:0/2165218182 wait complete. 2023-07-14T18:03:51.586 INFO:teuthology.orchestra.run.smithi115.stdout:mgr not available, waiting (2/15)... 2023-07-14T18:03:51.844 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:51 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:51.374+0000 7f0e0c119200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:03:52.596 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:03:52.235+0000 7f0e0c119200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:03:52.596 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: Activating manager daemon x 2023-07-14T18:03:52.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: mgrmap e2: x(active, starting, since 0.00315864s) 2023-07-14T18:03:52.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T18:03:52.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T18:03:52.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T18:03:52.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:03:52.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T18:03:52.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: Manager daemon x is now available 2023-07-14T18:03:52.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T18:03:52.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T18:03:52.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' 2023-07-14T18:03:52.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:52 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' 2023-07-14T18:03:53.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:53 smithi115 ceph-mon[127451]: from='mgr.14100 172.21.15.115:0/2198339502' entity='mgr.x' 2023-07-14T18:03:53.595 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:53 smithi115 ceph-mon[127451]: mgrmap e3: x(active, since 1.00747s) 2023-07-14T18:03:55.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:55 smithi115 ceph-mon[127451]: mgrmap e4: x(active, since 2s) 2023-07-14T18:03:56.594 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:56 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1459986198' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:03:58.327 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:58.327 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:03:58.327 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:03:58.328 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:03:58.328 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:03:58.328 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:03:58.328 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:03:58.328 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:58.328 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:03:58.328 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:03:58.329 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:03:58.329 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:03:58.329 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:03:58.329 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:03:58.329 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:03:58.329 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2023-07-14T18:03:58.330 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:03:58.330 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:58.331 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:03:58.331 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:03:58.331 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:58.331 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:03:58.331 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:58.331 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:03:58.331 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:03:58.331 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:03:58.337 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:03:58.338 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:03:58.338 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:03:58.338 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:58.338 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:03:58.338 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:03:58.338 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:03:58.338 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:03:58.339 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:03:58.339 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:03:58.339 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:03:58.339 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:03:58.339 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:03:58.339 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:58.339 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:03:58.340 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:58.340 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:03:58.340 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:03:58.340 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:58.340 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:03:58.340 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T18:03:58.340 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:03:58.340 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:03:58.341 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:03:58.341 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:03:58.341 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:03:58.341 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:03:58.341 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:03:58.341 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:58.341 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:03:58.341 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:03:58.342 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:03:38.101692+0000", 2023-07-14T18:03:58.342 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:03:58.342 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:03:58.342 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:03:58.342 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:03:58.342 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.939+0000 7f2fe3a5b700 1 Processor -- start 2023-07-14T18:03:58.342 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.939+0000 7f2fe3a5b700 1 -- start start 2023-07-14T18:03:58.343 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.939+0000 7f2fe3a5b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc1055a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:58.343 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.939+0000 7f2fe3a5b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc105ae0 con 0x7f2fdc105190 2023-07-14T18:03:58.343 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.940+0000 7f2fe17f7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc1055a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:58.343 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.940+0000 7f2fe17f7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc1055a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51266/0 (socket says 172.21.15.115:51266) 2023-07-14T18:03:58.343 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.940+0000 7f2fe17f7700 1 -- 172.21.15.115:0/3759770573 learned_addr learned my addr 172.21.15.115:0/3759770573 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:58.343 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.940+0000 7f2fe17f7700 1 -- 172.21.15.115:0/3759770573 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc105c20 con 0x7f2fdc105190 2023-07-14T18:03:58.343 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.940+0000 7f2fe17f7700 1 --2- 172.21.15.115:0/3759770573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc1055a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0098a0 tx=0x7f2fcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9268c801f0dde4a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:58.344 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.940+0000 7f2fd3fff700 1 -- 172.21.15.115:0/3759770573 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fcc004160 con 0x7f2fdc105190 2023-07-14T18:03:58.344 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.941+0000 7f2fd3fff700 1 -- 172.21.15.115:0/3759770573 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2fcc0042c0 con 0x7f2fdc105190 2023-07-14T18:03:58.344 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.941+0000 7f2fd3fff700 1 -- 172.21.15.115:0/3759770573 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fcc011940 con 0x7f2fdc105190 2023-07-14T18:03:58.344 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.941+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/3759770573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 msgr2=0x7f2fdc1055a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:58.344 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.941+0000 7f2fe3a5b700 1 --2- 172.21.15.115:0/3759770573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc1055a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc0098a0 tx=0x7f2fcc009bb0 comp rx=0 tx=0).stop 2023-07-14T18:03:58.344 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.941+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/3759770573 shutdown_connections 2023-07-14T18:03:58.344 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.941+0000 7f2fe3a5b700 1 --2- 172.21.15.115:0/3759770573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc1055a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:58.344 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.941+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/3759770573 >> 172.21.15.115:0/3759770573 conn(0x7f2fdc100820 msgr2=0x7f2fdc102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:58.345 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.941+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/3759770573 shutdown_connections 2023-07-14T18:03:58.345 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.941+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/3759770573 wait complete. 2023-07-14T18:03:58.345 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.942+0000 7f2fe3a5b700 1 Processor -- start 2023-07-14T18:03:58.345 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.942+0000 7f2fe3a5b700 1 -- start start 2023-07-14T18:03:58.345 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.942+0000 7f2fe3a5b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc19af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:58.346 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.942+0000 7f2fe3a5b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc19b450 con 0x7f2fdc105190 2023-07-14T18:03:58.346 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.942+0000 7f2fe17f7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc19af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:58.346 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.942+0000 7f2fe17f7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc19af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51268/0 (socket says 172.21.15.115:51268) 2023-07-14T18:03:58.346 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.942+0000 7f2fe17f7700 1 -- 172.21.15.115:0/1459986198 learned_addr learned my addr 172.21.15.115:0/1459986198 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:58.346 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.942+0000 7f2fe17f7700 1 -- 172.21.15.115:0/1459986198 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc009550 con 0x7f2fdc105190 2023-07-14T18:03:58.346 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.942+0000 7f2fe17f7700 1 --2- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc19af10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc006af0 tx=0x7f2fcc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:58.346 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.943+0000 7f2fd27fc700 1 -- 172.21.15.115:0/1459986198 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fcc003bf0 con 0x7f2fdc105190 2023-07-14T18:03:58.347 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.943+0000 7f2fd27fc700 1 -- 172.21.15.115:0/1459986198 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2fcc01b8b0 con 0x7f2fdc105190 2023-07-14T18:03:58.347 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.943+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc19b650 con 0x7f2fdc105190 2023-07-14T18:03:58.347 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.943+0000 7f2fd27fc700 1 -- 172.21.15.115:0/1459986198 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fcc019880 con 0x7f2fdc105190 2023-07-14T18:03:58.347 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.943+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc19ba70 con 0x7f2fdc105190 2023-07-14T18:03:58.347 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.943+0000 7f2fd27fc700 1 -- 172.21.15.115:0/1459986198 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f2fcc019b40 con 0x7f2fdc105190 2023-07-14T18:03:58.347 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.944+0000 7f2fd27fc700 1 --2- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f2fc8038730 0x7f2fc803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:58.347 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.944+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc061c70 con 0x7f2fdc105190 2023-07-14T18:03:58.347 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.944+0000 7f2fd27fc700 1 -- 172.21.15.115:0/1459986198 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2fcc04cbb0 con 0x7f2fdc105190 2023-07-14T18:03:58.348 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.944+0000 7f2fe0ff6700 1 --2- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f2fc8038730 0x7f2fc803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:58.348 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.944+0000 7f2fe0ff6700 1 --2- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f2fc8038730 0x7f2fc803abe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8006fd0 tx=0x7f2fd8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:58.348 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:55.949+0000 7f2fd27fc700 1 -- 172.21.15.115:0/1459986198 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2fcc04ee60 con 0x7f2fdc105190 2023-07-14T18:03:58.348 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.210+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2fdc19eb70 con 0x7f2fdc105190 2023-07-14T18:03:58.348 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.210+0000 7f2fd27fc700 1 -- 172.21.15.115:0/1459986198 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f2fcc018530 con 0x7f2fdc105190 2023-07-14T18:03:58.348 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.212+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f2fc8038730 msgr2=0x7f2fc803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:58.348 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.212+0000 7f2fe3a5b700 1 --2- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f2fc8038730 0x7f2fc803abe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8006fd0 tx=0x7f2fd8006e40 comp rx=0 tx=0).stop 2023-07-14T18:03:58.349 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.212+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 msgr2=0x7f2fdc19af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:58.349 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.212+0000 7f2fe3a5b700 1 --2- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc19af10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc006af0 tx=0x7f2fcc011a30 comp rx=0 tx=0).stop 2023-07-14T18:03:58.349 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.212+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 shutdown_connections 2023-07-14T18:03:58.349 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.212+0000 7f2fe3a5b700 1 --2- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f2fc8038730 0x7f2fc803abe0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:58.349 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.212+0000 7f2fe3a5b700 1 --2- 172.21.15.115:0/1459986198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc105190 0x7f2fdc19af10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:58.349 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.212+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 >> 172.21.15.115:0/1459986198 conn(0x7f2fdc100820 msgr2=0x7f2fdc101490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:58.349 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.213+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 shutdown_connections 2023-07-14T18:03:58.349 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:56.213+0000 7f2fe3a5b700 1 -- 172.21.15.115:0/1459986198 wait complete. 2023-07-14T18:03:58.350 INFO:teuthology.orchestra.run.smithi115.stdout:mgr is available 2023-07-14T18:03:59.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:03:59 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2345795235' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T18:03:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T18:03:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout fsid = a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:03:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T18:03:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T18:03:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T18:03:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T18:03:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T18:03:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T18:03:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 2023-07-14T18:03:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T18:03:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T18:03:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T18:03:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T18:03:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T18:03:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T18:03:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.805+0000 7f6240aac700 1 Processor -- start 2023-07-14T18:03:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.805+0000 7f6240aac700 1 -- start start 2023-07-14T18:03:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.805+0000 7f6240aac700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c105ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.805+0000 7f6240aac700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c105ff0 con 0x7f623c1056a0 2023-07-14T18:03:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.805+0000 7f623a59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c105ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.805+0000 7f623a59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c105ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51272/0 (socket says 172.21.15.115:51272) 2023-07-14T18:03:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.805+0000 7f623a59c700 1 -- 172.21.15.115:0/998851712 learned_addr learned my addr 172.21.15.115:0/998851712 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.805+0000 7f623a59c700 1 -- 172.21.15.115:0/998851712 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c106130 con 0x7f623c1056a0 2023-07-14T18:03:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.806+0000 7f623a59c700 1 --2- 172.21.15.115:0/998851712 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c105ab0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f622c0098a0 tx=0x7f622c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c717177c5db9384 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.806+0000 7f623959a700 1 -- 172.21.15.115:0/998851712 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f622c004160 con 0x7f623c1056a0 2023-07-14T18:03:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.806+0000 7f623959a700 1 -- 172.21.15.115:0/998851712 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f622c0042c0 con 0x7f623c1056a0 2023-07-14T18:03:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.806+0000 7f623959a700 1 -- 172.21.15.115:0/998851712 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f622c011940 con 0x7f623c1056a0 2023-07-14T18:03:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.806+0000 7f6240aac700 1 -- 172.21.15.115:0/998851712 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 msgr2=0x7f623c105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.806+0000 7f6240aac700 1 --2- 172.21.15.115:0/998851712 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c105ab0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f622c0098a0 tx=0x7f622c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:03:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.806+0000 7f6240aac700 1 -- 172.21.15.115:0/998851712 shutdown_connections 2023-07-14T18:03:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.806+0000 7f6240aac700 1 --2- 172.21.15.115:0/998851712 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c105ab0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.806+0000 7f6240aac700 1 -- 172.21.15.115:0/998851712 >> 172.21.15.115:0/998851712 conn(0x7f623c0fee20 msgr2=0x7f623c101230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.807+0000 7f6240aac700 1 -- 172.21.15.115:0/998851712 shutdown_connections 2023-07-14T18:03:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.807+0000 7f6240aac700 1 -- 172.21.15.115:0/998851712 wait complete. 2023-07-14T18:03:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.807+0000 7f6240aac700 1 Processor -- start 2023-07-14T18:03:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.807+0000 7f6240aac700 1 -- start start 2023-07-14T18:03:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.807+0000 7f6240aac700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c19ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.807+0000 7f6240aac700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623c19b370 con 0x7f623c1056a0 2023-07-14T18:03:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f623a59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c19ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f623a59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c19ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51274/0 (socket says 172.21.15.115:51274) 2023-07-14T18:03:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f623a59c700 1 -- 172.21.15.115:0/2345795235 learned_addr learned my addr 172.21.15.115:0/2345795235 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:03:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f623a59c700 1 -- 172.21.15.115:0/2345795235 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c009550 con 0x7f623c1056a0 2023-07-14T18:03:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f623a59c700 1 --2- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c19ae30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f622c011dc0 tx=0x7f622c011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f622b7fe700 1 -- 172.21.15.115:0/2345795235 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f622c0037d0 con 0x7f623c1056a0 2023-07-14T18:03:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f622b7fe700 1 -- 172.21.15.115:0/2345795235 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f622c01b440 con 0x7f623c1056a0 2023-07-14T18:03:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623c19b570 con 0x7f623c1056a0 2023-07-14T18:03:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f622b7fe700 1 -- 172.21.15.115:0/2345795235 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f622c01a560 con 0x7f623c1056a0 2023-07-14T18:03:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.808+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623c19b990 con 0x7f623c1056a0 2023-07-14T18:03:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.809+0000 7f622b7fe700 1 -- 172.21.15.115:0/2345795235 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f622c01a7d0 con 0x7f623c1056a0 2023-07-14T18:03:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.809+0000 7f622b7fe700 1 --2- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f62240387d0 0x7f622403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:03:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.809+0000 7f622b7fe700 1 -- 172.21.15.115:0/2345795235 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f622c04c800 con 0x7f623c1056a0 2023-07-14T18:03:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.809+0000 7f6239d9b700 1 --2- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f62240387d0 0x7f622403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:03:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.809+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623c061c70 con 0x7f623c1056a0 2023-07-14T18:03:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.810+0000 7f6239d9b700 1 --2- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f62240387d0 0x7f622403ac80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6230006fd0 tx=0x7f6230006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:03:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.814+0000 7f622b7fe700 1 -- 172.21.15.115:0/2345795235 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f622c023590 con 0x7f623c1056a0 2023-07-14T18:03:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.995+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f623c19e3d0 con 0x7f623c1056a0 2023-07-14T18:03:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.996+0000 7f622b7fe700 1 -- 172.21.15.115:0/2345795235 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f622c0233b0 con 0x7f623c1056a0 2023-07-14T18:03:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.998+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f62240387d0 msgr2=0x7f622403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.998+0000 7f6240aac700 1 --2- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f62240387d0 0x7f622403ac80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6230006fd0 tx=0x7f6230006e40 comp rx=0 tx=0).stop 2023-07-14T18:03:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.998+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 msgr2=0x7f623c19ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:03:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.998+0000 7f6240aac700 1 --2- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c19ae30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f622c011dc0 tx=0x7f622c011ea0 comp rx=0 tx=0).stop 2023-07-14T18:03:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.998+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 shutdown_connections 2023-07-14T18:03:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.998+0000 7f6240aac700 1 --2- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f62240387d0 0x7f622403ac80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.998+0000 7f6240aac700 1 --2- 172.21.15.115:0/2345795235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f623c1056a0 0x7f623c19ae30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:03:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.998+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 >> 172.21.15.115:0/2345795235 conn(0x7f623c0fee20 msgr2=0x7f623c0ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:03:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.999+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 shutdown_connections 2023-07-14T18:03:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:58.999+0000 7f6240aac700 1 -- 172.21.15.115:0/2345795235 wait complete. 2023-07-14T18:03:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling cephadm module... 2023-07-14T18:04:00.466 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:00 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2160145315' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-14T18:04:00.467 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:00 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: ignoring --setuser ceph since I am not root 2023-07-14T18:04:00.467 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:00 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: ignoring --setgroup ceph since I am not root 2023-07-14T18:04:00.467 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:00 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:00.269+0000 7f0e96384200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:04:00.467 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:00 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:00.349+0000 7f0e96384200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:04:00.467 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:00 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:00.438+0000 7f0e96384200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:04:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.808+0000 7f6139722700 1 Processor -- start 2023-07-14T18:04:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.808+0000 7f6139722700 1 -- start start 2023-07-14T18:04:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.808+0000 7f6139722700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134105a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.808+0000 7f6139722700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6134105fa0 con 0x7f6134105650 2023-07-14T18:04:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.808+0000 7f6132ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134105a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.808+0000 7f6132ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134105a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51278/0 (socket says 172.21.15.115:51278) 2023-07-14T18:04:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.808+0000 7f6132ffd700 1 -- 172.21.15.115:0/3726220063 learned_addr learned my addr 172.21.15.115:0/3726220063 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.808+0000 7f6132ffd700 1 -- 172.21.15.115:0/3726220063 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61341060e0 con 0x7f6134105650 2023-07-14T18:04:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6132ffd700 1 --2- 172.21.15.115:0/3726220063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134105a60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61240098a0 tx=0x7f6124009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a1574a2650cf73 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6131ffb700 1 -- 172.21.15.115:0/3726220063 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6124004160 con 0x7f6134105650 2023-07-14T18:04:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6131ffb700 1 -- 172.21.15.115:0/3726220063 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f61240042c0 con 0x7f6134105650 2023-07-14T18:04:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6131ffb700 1 -- 172.21.15.115:0/3726220063 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6124011940 con 0x7f6134105650 2023-07-14T18:04:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6139722700 1 -- 172.21.15.115:0/3726220063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 msgr2=0x7f6134105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6139722700 1 --2- 172.21.15.115:0/3726220063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134105a60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61240098a0 tx=0x7f6124009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6139722700 1 -- 172.21.15.115:0/3726220063 shutdown_connections 2023-07-14T18:04:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6139722700 1 --2- 172.21.15.115:0/3726220063 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134105a60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6139722700 1 -- 172.21.15.115:0/3726220063 >> 172.21.15.115:0/3726220063 conn(0x7f61340fee80 msgr2=0x7f6134101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6139722700 1 -- 172.21.15.115:0/3726220063 shutdown_connections 2023-07-14T18:04:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.809+0000 7f6139722700 1 -- 172.21.15.115:0/3726220063 wait complete. 2023-07-14T18:04:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.810+0000 7f6139722700 1 Processor -- start 2023-07-14T18:04:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.810+0000 7f6139722700 1 -- start start 2023-07-14T18:04:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f6139722700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f6139722700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61341980c0 con 0x7f6134105650 2023-07-14T18:04:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f6132ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f6132ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51280/0 (socket says 172.21.15.115:51280) 2023-07-14T18:04:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f6132ffd700 1 -- 172.21.15.115:0/2160145315 learned_addr learned my addr 172.21.15.115:0/2160145315 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f6132ffd700 1 -- 172.21.15.115:0/2160145315 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6124009550 con 0x7f6134105650 2023-07-14T18:04:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f6132ffd700 1 --2- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134197b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6124006af0 tx=0x7f6124011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f611bfff700 1 -- 172.21.15.115:0/2160145315 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6124003bf0 con 0x7f6134105650 2023-07-14T18:04:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61341982c0 con 0x7f6134105650 2023-07-14T18:04:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f611bfff700 1 -- 172.21.15.115:0/2160145315 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f612401b8b0 con 0x7f6134105650 2023-07-14T18:04:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f611bfff700 1 -- 172.21.15.115:0/2160145315 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6124019880 con 0x7f6134105650 2023-07-14T18:04:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.811+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61341986e0 con 0x7f6134105650 2023-07-14T18:04:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.812+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6134061c70 con 0x7f6134105650 2023-07-14T18:04:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.812+0000 7f611bfff700 1 -- 172.21.15.115:0/2160145315 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f6124019b40 con 0x7f6134105650 2023-07-14T18:04:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.812+0000 7f611bfff700 1 --2- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f611c038730 0x7f611c03abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.812+0000 7f611bfff700 1 -- 172.21.15.115:0/2160145315 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f612404cbb0 con 0x7f6134105650 2023-07-14T18:04:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.813+0000 7f61327fc700 1 --2- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f611c038730 0x7f611c03abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.813+0000 7f61327fc700 1 --2- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f611c038730 0x7f611c03abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6128006fd0 tx=0x7f6128006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:03:59.817+0000 7f611bfff700 1 -- 172.21.15.115:0/2160145315 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f612401a940 con 0x7f6134105650 2023-07-14T18:04:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.053+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f613419b870 con 0x7f6134105650 2023-07-14T18:04:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.105+0000 7f611bfff700 1 -- 172.21.15.115:0/2160145315 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f612401a3d0 con 0x7f6134105650 2023-07-14T18:04:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.106+0000 7f611bfff700 1 -- 172.21.15.115:0/2160145315 <== mon.0 v2:172.21.15.115:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f612404c5d0 con 0x7f6134105650 2023-07-14T18:04:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f611c038730 msgr2=0x7f611c03abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 --2- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f611c038730 0x7f611c03abe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6128006fd0 tx=0x7f6128006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 msgr2=0x7f6134197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 --2- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134197b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6124006af0 tx=0x7f6124011a30 comp rx=0 tx=0).stop 2023-07-14T18:04:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 shutdown_connections 2023-07-14T18:04:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 --2- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f611c038730 0x7f611c03abe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 --2- 172.21.15.115:0/2160145315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6134105650 0x7f6134197b80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 >> 172.21.15.115:0/2160145315 conn(0x7f61340fee80 msgr2=0x7f61340ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 shutdown_connections 2023-07-14T18:04:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:00.109+0000 7f6139722700 1 -- 172.21.15.115:0/2160145315 wait complete. 2023-07-14T18:04:00.740 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:00 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:00.532+0000 7f0e96384200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:04:00.740 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:00 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:00.711+0000 7f0e96384200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:04:01.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:01 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2160145315' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-14T18:04:01.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:01 smithi115 ceph-mon[127451]: mgrmap e5: x(active, since 7s) 2023-07-14T18:04:01.345 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:01 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:01.036+0000 7f0e96384200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:04:01.345 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:01 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:01.131+0000 7f0e96384200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:04:01.346 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:01 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:01.261+0000 7f0e96384200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:04:01.744 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:01 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:01.359+0000 7f0e96384200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:04:01.772 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:04:01.772 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-14T18:04:01.772 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T18:04:01.772 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T18:04:01.773 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T18:04:01.773 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:04:01.773 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.083+0000 7fa2c1350700 1 Processor -- start 2023-07-14T18:04:01.773 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.083+0000 7fa2c1350700 1 -- start start 2023-07-14T18:04:01.773 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2c1350700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc1080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:01.773 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2c1350700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc079d10 con 0x7fa2bc105cd0 2023-07-14T18:04:01.773 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc1080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:01.774 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc1080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51294/0 (socket says 172.21.15.115:51294) 2023-07-14T18:04:01.774 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2baffd700 1 -- 172.21.15.115:0/1891788389 learned_addr learned my addr 172.21.15.115:0/1891788389 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:01.774 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2baffd700 1 -- 172.21.15.115:0/1891788389 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc079e50 con 0x7fa2bc105cd0 2023-07-14T18:04:01.774 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2baffd700 1 --2- 172.21.15.115:0/1891788389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc1080b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0098a0 tx=0x7fa2ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbfe120ddab13012 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:01.774 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2b9ffb700 1 -- 172.21.15.115:0/1891788389 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2ac004160 con 0x7fa2bc105cd0 2023-07-14T18:04:01.774 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2b9ffb700 1 -- 172.21.15.115:0/1891788389 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa2ac0042c0 con 0x7fa2bc105cd0 2023-07-14T18:04:01.774 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.084+0000 7fa2b9ffb700 1 -- 172.21.15.115:0/1891788389 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2ac011940 con 0x7fa2bc105cd0 2023-07-14T18:04:01.775 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.085+0000 7fa2c1350700 1 -- 172.21.15.115:0/1891788389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 msgr2=0x7fa2bc1080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:01.775 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.085+0000 7fa2c1350700 1 --2- 172.21.15.115:0/1891788389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc1080b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0098a0 tx=0x7fa2ac009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:01.775 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.085+0000 7fa2c1350700 1 -- 172.21.15.115:0/1891788389 shutdown_connections 2023-07-14T18:04:01.775 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.085+0000 7fa2c1350700 1 --2- 172.21.15.115:0/1891788389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc1080b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:01.775 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.085+0000 7fa2c1350700 1 -- 172.21.15.115:0/1891788389 >> 172.21.15.115:0/1891788389 conn(0x7fa2bc0ff710 msgr2=0x7fa2bc101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:01.775 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.085+0000 7fa2c1350700 1 -- 172.21.15.115:0/1891788389 shutdown_connections 2023-07-14T18:04:01.775 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.085+0000 7fa2c1350700 1 -- 172.21.15.115:0/1891788389 wait complete. 2023-07-14T18:04:01.776 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.086+0000 7fa2c1350700 1 Processor -- start 2023-07-14T18:04:01.776 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.086+0000 7fa2c1350700 1 -- start start 2023-07-14T18:04:01.776 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.086+0000 7fa2c1350700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:01.776 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.086+0000 7fa2c1350700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc105b70 con 0x7fa2bc105cd0 2023-07-14T18:04:01.776 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.086+0000 7fa2baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:01.776 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.086+0000 7fa2baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51296/0 (socket says 172.21.15.115:51296) 2023-07-14T18:04:01.776 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.086+0000 7fa2baffd700 1 -- 172.21.15.115:0/3091437008 learned_addr learned my addr 172.21.15.115:0/3091437008 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:01.777 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.086+0000 7fa2baffd700 1 -- 172.21.15.115:0/3091437008 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac009550 con 0x7fa2bc105cd0 2023-07-14T18:04:01.777 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.086+0000 7fa2baffd700 1 --2- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc105630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac000c00 tx=0x7fa2ac011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:01.777 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.087+0000 7fa2a3fff700 1 -- 172.21.15.115:0/3091437008 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2ac0036a0 con 0x7fa2bc105cd0 2023-07-14T18:04:01.777 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.087+0000 7fa2a3fff700 1 -- 172.21.15.115:0/3091437008 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa2ac01b440 con 0x7fa2bc105cd0 2023-07-14T18:04:01.777 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.087+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc103c80 con 0x7fa2bc105cd0 2023-07-14T18:04:01.777 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.087+0000 7fa2a3fff700 1 -- 172.21.15.115:0/3091437008 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2ac0193f0 con 0x7fa2bc105cd0 2023-07-14T18:04:01.777 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.087+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc104100 con 0x7fa2bc105cd0 2023-07-14T18:04:01.778 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.088+0000 7fa2a3fff700 1 -- 172.21.15.115:0/3091437008 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fa2ac019620 con 0x7fa2bc105cd0 2023-07-14T18:04:01.778 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.088+0000 7fa2a3fff700 1 --2- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7fa2a4038820 0x7fa2a403acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:01.778 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.088+0000 7fa2a3fff700 1 -- 172.21.15.115:0/3091437008 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa2ac04c5c0 con 0x7fa2bc105cd0 2023-07-14T18:04:01.778 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.088+0000 7fa2ba7fc700 1 -- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7fa2a4038820 msgr2=0x7fa2a403acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/568655687 2023-07-14T18:04:01.778 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.088+0000 7fa2ba7fc700 1 --2- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7fa2a4038820 0x7fa2a403acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:04:01.778 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.088+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc061c70 con 0x7fa2bc105cd0 2023-07-14T18:04:01.779 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.093+0000 7fa2a3fff700 1 -- 172.21.15.115:0/3091437008 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa2ac01f020 con 0x7fa2bc105cd0 2023-07-14T18:04:01.779 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.289+0000 7fa2ba7fc700 1 -- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7fa2a4038820 msgr2=0x7fa2a403acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/568655687 2023-07-14T18:04:01.779 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.289+0000 7fa2ba7fc700 1 --2- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7fa2a4038820 0x7fa2a403acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:04:01.779 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.337+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa2bc104b00 con 0x7fa2bc105cd0 2023-07-14T18:04:01.779 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.338+0000 7fa2a3fff700 1 -- 172.21.15.115:0/3091437008 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fa2ac01b940 con 0x7fa2bc105cd0 2023-07-14T18:04:01.780 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7fa2a4038820 msgr2=0x7fa2a403acd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:04:01.780 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 --2- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7fa2a4038820 0x7fa2a403acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:01.780 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 msgr2=0x7fa2bc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:01.780 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 --2- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc105630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac000c00 tx=0x7fa2ac011e70 comp rx=0 tx=0).stop 2023-07-14T18:04:01.781 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 shutdown_connections 2023-07-14T18:04:01.781 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 --2- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7fa2a4038820 0x7fa2a403acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:01.781 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 --2- 172.21.15.115:0/3091437008 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2bc105cd0 0x7fa2bc105630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:01.781 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 >> 172.21.15.115:0/3091437008 conn(0x7fa2bc0ff710 msgr2=0x7fa2bc1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:01.782 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 shutdown_connections 2023-07-14T18:04:01.782 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:01.340+0000 7fa2c1350700 1 -- 172.21.15.115:0/3091437008 wait complete. 2023-07-14T18:04:01.782 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for the mgr to restart... 2023-07-14T18:04:01.782 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mgr epoch 5... 2023-07-14T18:04:02.004 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:01 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:01.795+0000 7f0e96384200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:04:02.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:02 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3091437008' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T18:04:02.845 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:02 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:02.377+0000 7f0e96384200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:04:02.845 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:02 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:02.544+0000 7f0e96384200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:04:03.583 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:03.298+0000 7f0e96384200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:04:03.584 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:03.388+0000 7f0e96384200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:04:03.845 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:03.555+0000 7f0e96384200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:04:04.226 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:03.829+0000 7f0e96384200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:04:04.226 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:04 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:04.053+0000 7f0e96384200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:04:04.483 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:04 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:04.197+0000 7f0e96384200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:04:04.845 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:04 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:04.455+0000 7f0e96384200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:04:05.491 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:05 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:05.147+0000 7f0e96384200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:04:05.492 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:05 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:05.276+0000 7f0e96384200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:04:05.816 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:05 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:05.463+0000 7f0e96384200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:04:05.817 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:05 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:05.575+0000 7f0e96384200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:04:06.095 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:05 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:05.789+0000 7f0e96384200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:04:07.095 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:06.633+0000 7f0e96384200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:04:07.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: Active manager daemon x restarted 2023-07-14T18:04:07.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: Activating manager daemon x 2023-07-14T18:04:07.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: osdmap e2: 0 total, 0 up, 0 in 2023-07-14T18:04:07.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: mgrmap e6: x(active, starting, since 0.00337065s) 2023-07-14T18:04:07.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:04:07.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T18:04:07.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T18:04:07.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T18:04:07.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T18:04:07.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: Manager daemon x is now available 2023-07-14T18:04:07.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:07.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:07.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:07.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:07.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:06 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:08.029 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:04:08.029 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-14T18:04:08.029 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T18:04:08.030 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:04:08.030 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.276+0000 7f69469d0700 1 Processor -- start 2023-07-14T18:04:08.030 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.276+0000 7f69469d0700 1 -- start start 2023-07-14T18:04:08.030 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.276+0000 7f69469d0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6940079cd0 0x7f694010a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:08.030 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.276+0000 7f69469d0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6940105b10 con 0x7f6940079cd0 2023-07-14T18:04:08.030 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.276+0000 7f693ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6940079cd0 0x7f694010a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:08.030 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.276+0000 7f693ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6940079cd0 0x7f694010a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51304/0 (socket says 172.21.15.115:51304) 2023-07-14T18:04:08.031 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.276+0000 7f693ffff700 1 -- 172.21.15.115:0/1144529295 learned_addr learned my addr 172.21.15.115:0/1144529295 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:08.031 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.276+0000 7f693ffff700 1 -- 172.21.15.115:0/1144529295 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694010a660 con 0x7f6940079cd0 2023-07-14T18:04:08.031 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.276+0000 7f693ffff700 1 --2- 172.21.15.115:0/1144529295 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6940079cd0 0x7f694010a120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f69300098a0 tx=0x7f6930009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21bbd23add09aa5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:08.031 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f693effd700 1 -- 172.21.15.115:0/1144529295 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6930004160 con 0x7f6940079cd0 2023-07-14T18:04:08.031 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f693effd700 1 -- 172.21.15.115:0/1144529295 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f69300042c0 con 0x7f6940079cd0 2023-07-14T18:04:08.031 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f693effd700 1 -- 172.21.15.115:0/1144529295 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6930011940 con 0x7f6940079cd0 2023-07-14T18:04:08.032 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f69469d0700 1 -- 172.21.15.115:0/1144529295 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6940079cd0 msgr2=0x7f694010a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:08.032 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f69469d0700 1 --2- 172.21.15.115:0/1144529295 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6940079cd0 0x7f694010a120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f69300098a0 tx=0x7f6930009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:08.032 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f69469d0700 1 -- 172.21.15.115:0/1144529295 shutdown_connections 2023-07-14T18:04:08.032 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f69469d0700 1 --2- 172.21.15.115:0/1144529295 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6940079cd0 0x7f694010a120 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:08.032 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f69469d0700 1 -- 172.21.15.115:0/1144529295 >> 172.21.15.115:0/1144529295 conn(0x7f69400ff710 msgr2=0x7f6940101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:08.032 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f69469d0700 1 -- 172.21.15.115:0/1144529295 shutdown_connections 2023-07-14T18:04:08.032 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.277+0000 7f69469d0700 1 -- 172.21.15.115:0/1144529295 wait complete. 2023-07-14T18:04:08.033 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.278+0000 7f69469d0700 1 Processor -- start 2023-07-14T18:04:08.033 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.278+0000 7f69469d0700 1 -- start start 2023-07-14T18:04:08.033 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f69469d0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f694019af00 0x7f694019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:08.033 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f69469d0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694019b850 con 0x7f694019af00 2023-07-14T18:04:08.033 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f693ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f694019af00 0x7f694019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:08.033 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f693ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f694019af00 0x7f694019b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51306/0 (socket says 172.21.15.115:51306) 2023-07-14T18:04:08.033 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f693ffff700 1 -- 172.21.15.115:0/3003192941 learned_addr learned my addr 172.21.15.115:0/3003192941 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:08.034 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f693ffff700 1 -- 172.21.15.115:0/3003192941 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6930009550 con 0x7f694019af00 2023-07-14T18:04:08.034 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f693ffff700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f694019af00 0x7f694019b310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6930000c00 tx=0x7f6930011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:08.034 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69300036a0 con 0x7f694019af00 2023-07-14T18:04:08.034 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f693001b440 con 0x7f694019af00 2023-07-14T18:04:08.034 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f69469d0700 1 -- 172.21.15.115:0/3003192941 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694019ba50 con 0x7f694019af00 2023-07-14T18:04:08.034 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69300193f0 con 0x7f694019af00 2023-07-14T18:04:08.034 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.279+0000 7f69469d0700 1 -- 172.21.15.115:0/3003192941 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694019e6b0 con 0x7f694019af00 2023-07-14T18:04:08.034 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.280+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f6930019640 con 0x7f694019af00 2023-07-14T18:04:08.035 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.280+0000 7f693d7fa700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 0x7f692803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:08.035 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.280+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 --> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f692803b3e0 con 0x7f6928038820 2023-07-14T18:04:08.035 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.280+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f693004c5a0 con 0x7f694019af00 2023-07-14T18:04:08.035 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.280+0000 7f693f7fe700 1 -- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 msgr2=0x7f692803acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/568655687 2023-07-14T18:04:08.035 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.280+0000 7f693f7fe700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 0x7f692803acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:04:08.035 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.481+0000 7f693f7fe700 1 -- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 msgr2=0x7f692803acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/568655687 2023-07-14T18:04:08.035 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.481+0000 7f693f7fe700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 0x7f692803acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:04:08.036 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.882+0000 7f693f7fe700 1 -- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 msgr2=0x7f692803acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/568655687 2023-07-14T18:04:08.036 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:02.882+0000 7f693f7fe700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 0x7f692803acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T18:04:08.036 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:03.683+0000 7f693f7fe700 1 -- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 msgr2=0x7f692803acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/568655687 2023-07-14T18:04:08.036 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:03.683+0000 7f693f7fe700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 0x7f692803acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T18:04:08.036 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:05.284+0000 7f693f7fe700 1 -- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 msgr2=0x7f692803acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/568655687 2023-07-14T18:04:08.036 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:05.284+0000 7f693f7fe700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 0x7f692803acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T18:04:08.036 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:06.649+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7f6930018c70 con 0x7f694019af00 2023-07-14T18:04:08.037 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:06.649+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 msgr2=0x7f692803acd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:04:08.037 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:06.649+0000 7f693d7fa700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 0x7f692803acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:08.037 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.650+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f693004de70 con 0x7f694019af00 2023-07-14T18:04:08.037 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.650+0000 7f693d7fa700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f692803bfb0 0x7f692803e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:08.037 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.650+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f692803b3e0 con 0x7f692803bfb0 2023-07-14T18:04:08.037 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.651+0000 7f693f7fe700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f692803bfb0 0x7f692803e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:08.037 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.651+0000 7f693f7fe700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f692803bfb0 0x7f692803e390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6934003a10 tx=0x7f69340092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:08.037 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.652+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 <== mgr.14118 v2:172.21.15.115:6800/2169355495 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f692803b3e0 con 0x7f692803bfb0 2023-07-14T18:04:08.038 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.658+0000 7f69469d0700 1 -- 172.21.15.115:0/3003192941 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f694019c280 con 0x7f692803bfb0 2023-07-14T18:04:08.038 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.658+0000 7f693d7fa700 1 -- 172.21.15.115:0/3003192941 <== mgr.14118 v2:172.21.15.115:6800/2169355495 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f694019c280 con 0x7f692803bfb0 2023-07-14T18:04:08.038 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 -- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f692803bfb0 msgr2=0x7f692803e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:08.038 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f692803bfb0 0x7f692803e390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6934003a10 tx=0x7f69340092b0 comp rx=0 tx=0).stop 2023-07-14T18:04:08.038 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 -- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f694019af00 msgr2=0x7f694019b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:08.038 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f694019af00 0x7f694019b310 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6930000c00 tx=0x7f6930011e70 comp rx=0 tx=0).stop 2023-07-14T18:04:08.038 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 -- 172.21.15.115:0/3003192941 shutdown_connections 2023-07-14T18:04:08.039 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f692803bfb0 0x7f692803e390 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:08.039 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f694019af00 0x7f694019b310 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:08.039 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 --2- 172.21.15.115:0/3003192941 >> [v2:172.21.15.115:6800/568655687,v1:172.21.15.115:6801/568655687] conn(0x7f6928038820 0x7f692803acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:08.039 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 -- 172.21.15.115:0/3003192941 >> 172.21.15.115:0/3003192941 conn(0x7f69400ff710 msgr2=0x7f69401003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:08.039 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 -- 172.21.15.115:0/3003192941 shutdown_connections 2023-07-14T18:04:08.039 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:07.659+0000 7f69469d0700 1 -- 172.21.15.115:0/3003192941 wait complete. 2023-07-14T18:04:08.039 INFO:teuthology.orchestra.run.smithi115.stdout:mgr epoch 5 is available 2023-07-14T18:04:08.039 INFO:teuthology.orchestra.run.smithi115.stdout:Setting orchestrator backend to cephadm... 2023-07-14T18:04:08.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:07 smithi115 ceph-mon[127451]: Found migration_current of "None". Setting to last migration. 2023-07-14T18:04:08.048 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:07 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T18:04:08.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:07 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T18:04:08.049 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:07 smithi115 ceph-mon[127451]: mgrmap e7: x(active, since 1.00534s) 2023-07-14T18:04:09.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:08 smithi115 ceph-mon[127451]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T18:04:09.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:08 smithi115 ceph-mon[127451]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T18:04:09.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:08 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:09.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:08 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:09.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:08 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:09.175 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.509+0000 7f46ceab7700 1 Processor -- start 2023-07-14T18:04:09.175 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.509+0000 7f46ceab7700 1 -- start start 2023-07-14T18:04:09.176 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.509+0000 7f46ceab7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8107d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:09.176 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.509+0000 7f46ceab7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8108270 con 0x7f46c8079cb0 2023-07-14T18:04:09.176 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.509+0000 7f46cc853700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8107d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:09.176 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.509+0000 7f46cc853700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8107d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51356/0 (socket says 172.21.15.115:51356) 2023-07-14T18:04:09.176 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.509+0000 7f46cc853700 1 -- 172.21.15.115:0/216319683 learned_addr learned my addr 172.21.15.115:0/216319683 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:09.176 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.509+0000 7f46cc853700 1 -- 172.21.15.115:0/216319683 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c81083b0 con 0x7f46c8079cb0 2023-07-14T18:04:09.176 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.509+0000 7f46cc853700 1 --2- 172.21.15.115:0/216319683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8107d30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0098a0 tx=0x7f46bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87a5f4fd227f57c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:09.177 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46c77fe700 1 -- 172.21.15.115:0/216319683 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46bc004160 con 0x7f46c8079cb0 2023-07-14T18:04:09.177 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46c77fe700 1 -- 172.21.15.115:0/216319683 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f46bc0042c0 con 0x7f46c8079cb0 2023-07-14T18:04:09.177 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46c77fe700 1 -- 172.21.15.115:0/216319683 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46bc011940 con 0x7f46c8079cb0 2023-07-14T18:04:09.177 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46ceab7700 1 -- 172.21.15.115:0/216319683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 msgr2=0x7f46c8107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:09.177 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46ceab7700 1 --2- 172.21.15.115:0/216319683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8107d30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0098a0 tx=0x7f46bc009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:09.177 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46ceab7700 1 -- 172.21.15.115:0/216319683 shutdown_connections 2023-07-14T18:04:09.177 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46ceab7700 1 --2- 172.21.15.115:0/216319683 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8107d30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:09.178 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46ceab7700 1 -- 172.21.15.115:0/216319683 >> 172.21.15.115:0/216319683 conn(0x7f46c80ff6e0 msgr2=0x7f46c8101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:09.178 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46ceab7700 1 -- 172.21.15.115:0/216319683 shutdown_connections 2023-07-14T18:04:09.178 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.510+0000 7f46ceab7700 1 -- 172.21.15.115:0/216319683 wait complete. 2023-07-14T18:04:09.178 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.511+0000 7f46ceab7700 1 Processor -- start 2023-07-14T18:04:09.178 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.511+0000 7f46ceab7700 1 -- start start 2023-07-14T18:04:09.178 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.511+0000 7f46ceab7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:09.178 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.511+0000 7f46ceab7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8199240 con 0x7f46c8079cb0 2023-07-14T18:04:09.178 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.511+0000 7f46cc853700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:09.179 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.511+0000 7f46cc853700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8198d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51358/0 (socket says 172.21.15.115:51358) 2023-07-14T18:04:09.179 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.511+0000 7f46cc853700 1 -- 172.21.15.115:0/3084350491 learned_addr learned my addr 172.21.15.115:0/3084350491 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:09.179 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.511+0000 7f46cc853700 1 -- 172.21.15.115:0/3084350491 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc009550 con 0x7f46c8079cb0 2023-07-14T18:04:09.179 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.511+0000 7f46cc853700 1 --2- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8198d00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46bc006af0 tx=0x7f46bc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:09.179 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.512+0000 7f46c5ffb700 1 -- 172.21.15.115:0/3084350491 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46bc003c30 con 0x7f46c8079cb0 2023-07-14T18:04:09.179 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.512+0000 7f46c5ffb700 1 -- 172.21.15.115:0/3084350491 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f46bc01b8b0 con 0x7f46c8079cb0 2023-07-14T18:04:09.179 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.512+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c8199440 con 0x7f46c8079cb0 2023-07-14T18:04:09.179 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.512+0000 7f46c5ffb700 1 -- 172.21.15.115:0/3084350491 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46bc0237d0 con 0x7f46c8079cb0 2023-07-14T18:04:09.180 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.512+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c8199860 con 0x7f46c8079cb0 2023-07-14T18:04:09.180 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.513+0000 7f46c5ffb700 1 -- 172.21.15.115:0/3084350491 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f46bc023a40 con 0x7f46c8079cb0 2023-07-14T18:04:09.180 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.513+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c8061c70 con 0x7f46c8079cb0 2023-07-14T18:04:09.180 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.513+0000 7f46c5ffb700 1 --2- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f46b80386b0 0x7f46b803ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:09.180 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.513+0000 7f46c5ffb700 1 -- 172.21.15.115:0/3084350491 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f46bc04cbf0 con 0x7f46c8079cb0 2023-07-14T18:04:09.180 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.513+0000 7f46c7fff700 1 --2- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f46b80386b0 0x7f46b803ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:09.180 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.513+0000 7f46c7fff700 1 --2- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f46b80386b0 0x7f46b803ab60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f46b4006fd0 tx=0x7f46b4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:09.181 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.518+0000 7f46c5ffb700 1 -- 172.21.15.115:0/3084350491 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f46bc023cf0 con 0x7f46c8079cb0 2023-07-14T18:04:09.181 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.706+0000 7f46c5ffb700 1 -- 172.21.15.115:0/3084350491 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f46bc01ba90 con 0x7f46c8079cb0 2023-07-14T18:04:09.181 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.719+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f46c8100a40 con 0x7f46b80386b0 2023-07-14T18:04:09.181 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.724+0000 7f46c5ffb700 1 -- 172.21.15.115:0/3084350491 <== mgr.14118 v2:172.21.15.115:6800/2169355495 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f46c8100a40 con 0x7f46b80386b0 2023-07-14T18:04:09.181 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.726+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f46b80386b0 msgr2=0x7f46b803ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:09.181 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.726+0000 7f46ceab7700 1 --2- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f46b80386b0 0x7f46b803ab60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f46b4006fd0 tx=0x7f46b4006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:09.181 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.726+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 msgr2=0x7f46c8198d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:09.181 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.726+0000 7f46ceab7700 1 --2- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8198d00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46bc006af0 tx=0x7f46bc011a30 comp rx=0 tx=0).stop 2023-07-14T18:04:09.182 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.726+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 shutdown_connections 2023-07-14T18:04:09.182 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.726+0000 7f46ceab7700 1 --2- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f46b80386b0 0x7f46b803ab60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:09.182 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.726+0000 7f46ceab7700 1 --2- 172.21.15.115:0/3084350491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f46c8079cb0 0x7f46c8198d00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:09.182 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.726+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 >> 172.21.15.115:0/3084350491 conn(0x7f46c80ff6e0 msgr2=0x7f46c8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:09.182 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.727+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 shutdown_connections 2023-07-14T18:04:09.182 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:08.727+0000 7f46ceab7700 1 -- 172.21.15.115:0/3084350491 wait complete. 2023-07-14T18:04:10.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:09 smithi115 ceph-mon[127451]: [14/Jul/2023:18:04:08] ENGINE Bus STARTING 2023-07-14T18:04:10.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:09 smithi115 ceph-mon[127451]: [14/Jul/2023:18:04:08] ENGINE Serving on http://172.21.15.115:8765 2023-07-14T18:04:10.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:09 smithi115 ceph-mon[127451]: [14/Jul/2023:18:04:08] ENGINE Serving on https://172.21.15.115:7150 2023-07-14T18:04:10.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:09 smithi115 ceph-mon[127451]: [14/Jul/2023:18:04:08] ENGINE Bus STARTED 2023-07-14T18:04:10.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:09 smithi115 ceph-mon[127451]: mgrmap e8: x(active, since 2s) 2023-07-14T18:04:10.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:09 smithi115 ceph-mon[127451]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:10.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:09 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:10.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:09 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:10.220 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-14T18:04:10.220 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.603+0000 7f29eaf6e700 1 Processor -- start 2023-07-14T18:04:10.220 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.603+0000 7f29eaf6e700 1 -- start start 2023-07-14T18:04:10.220 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.603+0000 7f29eaf6e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e4079cd0 0x7f29e410a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:10.221 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.603+0000 7f29eaf6e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e4105b10 con 0x7f29e4079cd0 2023-07-14T18:04:10.221 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.604+0000 7f29e8d0a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e4079cd0 0x7f29e410a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:10.221 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.604+0000 7f29e8d0a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e4079cd0 0x7f29e410a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51364/0 (socket says 172.21.15.115:51364) 2023-07-14T18:04:10.221 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.604+0000 7f29e8d0a700 1 -- 172.21.15.115:0/2389893478 learned_addr learned my addr 172.21.15.115:0/2389893478 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:10.221 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.604+0000 7f29e8d0a700 1 -- 172.21.15.115:0/2389893478 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e410a660 con 0x7f29e4079cd0 2023-07-14T18:04:10.221 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.604+0000 7f29e8d0a700 1 --2- 172.21.15.115:0/2389893478 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e4079cd0 0x7f29e410a120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29d80098a0 tx=0x7f29d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d2dfd5d97f8ad2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:10.221 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.604+0000 7f29e37fe700 1 -- 172.21.15.115:0/2389893478 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29d8004160 con 0x7f29e4079cd0 2023-07-14T18:04:10.221 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.604+0000 7f29e37fe700 1 -- 172.21.15.115:0/2389893478 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f29d80042c0 con 0x7f29e4079cd0 2023-07-14T18:04:10.222 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.604+0000 7f29e37fe700 1 -- 172.21.15.115:0/2389893478 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29d8011940 con 0x7f29e4079cd0 2023-07-14T18:04:10.222 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.605+0000 7f29eaf6e700 1 -- 172.21.15.115:0/2389893478 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e4079cd0 msgr2=0x7f29e410a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:10.222 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.605+0000 7f29eaf6e700 1 --2- 172.21.15.115:0/2389893478 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e4079cd0 0x7f29e410a120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29d80098a0 tx=0x7f29d8009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:10.222 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.605+0000 7f29eaf6e700 1 -- 172.21.15.115:0/2389893478 shutdown_connections 2023-07-14T18:04:10.222 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.605+0000 7f29eaf6e700 1 --2- 172.21.15.115:0/2389893478 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e4079cd0 0x7f29e410a120 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:10.222 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.605+0000 7f29eaf6e700 1 -- 172.21.15.115:0/2389893478 >> 172.21.15.115:0/2389893478 conn(0x7f29e40ff710 msgr2=0x7f29e4101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:10.223 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.605+0000 7f29eaf6e700 1 -- 172.21.15.115:0/2389893478 shutdown_connections 2023-07-14T18:04:10.223 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.605+0000 7f29eaf6e700 1 -- 172.21.15.115:0/2389893478 wait complete. 2023-07-14T18:04:10.223 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.605+0000 7f29eaf6e700 1 Processor -- start 2023-07-14T18:04:10.223 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.605+0000 7f29eaf6e700 1 -- start start 2023-07-14T18:04:10.223 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29eaf6e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e419af00 0x7f29e419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:10.223 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29eaf6e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e419b850 con 0x7f29e419af00 2023-07-14T18:04:10.223 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29e8d0a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e419af00 0x7f29e419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:10.223 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29e8d0a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e419af00 0x7f29e419b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51366/0 (socket says 172.21.15.115:51366) 2023-07-14T18:04:10.224 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29e8d0a700 1 -- 172.21.15.115:0/4031952190 learned_addr learned my addr 172.21.15.115:0/4031952190 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:10.224 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29e8d0a700 1 -- 172.21.15.115:0/4031952190 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d8009550 con 0x7f29e419af00 2023-07-14T18:04:10.224 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29e8d0a700 1 --2- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e419af00 0x7f29e419b310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f29d8000c00 tx=0x7f29d8011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:10.224 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29e1ffb700 1 -- 172.21.15.115:0/4031952190 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29d80036a0 con 0x7f29e419af00 2023-07-14T18:04:10.224 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29e1ffb700 1 -- 172.21.15.115:0/4031952190 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f29d801b440 con 0x7f29e419af00 2023-07-14T18:04:10.224 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e419ba50 con 0x7f29e419af00 2023-07-14T18:04:10.224 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29e1ffb700 1 -- 172.21.15.115:0/4031952190 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29d80193f0 con 0x7f29e419af00 2023-07-14T18:04:10.225 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.606+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e419e6b0 con 0x7f29e419af00 2023-07-14T18:04:10.225 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.607+0000 7f29e1ffb700 1 -- 172.21.15.115:0/4031952190 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f29d8019630 con 0x7f29e419af00 2023-07-14T18:04:10.225 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.607+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e4079cd0 con 0x7f29e419af00 2023-07-14T18:04:10.225 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.607+0000 7f29e1ffb700 1 --2- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f29d40387d0 0x7f29d403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:10.225 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.607+0000 7f29e1ffb700 1 -- 172.21.15.115:0/4031952190 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f29d804c640 con 0x7f29e419af00 2023-07-14T18:04:10.225 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.607+0000 7f29e3fff700 1 --2- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f29d40387d0 0x7f29d403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:10.225 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.608+0000 7f29e3fff700 1 --2- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f29d40387d0 0x7f29d403ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29d0006fd0 tx=0x7f29d0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:10.225 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.612+0000 7f29e1ffb700 1 -- 172.21.15.115:0/4031952190 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f29d801a3d0 con 0x7f29e419af00 2023-07-14T18:04:10.226 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.800+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f29e4100b00 con 0x7f29d40387d0 2023-07-14T18:04:10.226 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.800+0000 7f29e1ffb700 1 -- 172.21.15.115:0/4031952190 <== mgr.14118 v2:172.21.15.115:6800/2169355495 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f29e4100b00 con 0x7f29d40387d0 2023-07-14T18:04:10.226 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f29d40387d0 msgr2=0x7f29d403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:10.226 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 --2- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f29d40387d0 0x7f29d403ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29d0006fd0 tx=0x7f29d0006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:10.226 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e419af00 msgr2=0x7f29e419b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:10.226 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 --2- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e419af00 0x7f29e419b310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f29d8000c00 tx=0x7f29d8011e70 comp rx=0 tx=0).stop 2023-07-14T18:04:10.226 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 shutdown_connections 2023-07-14T18:04:10.226 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 --2- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f29d40387d0 0x7f29d403ac80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:10.227 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 --2- 172.21.15.115:0/4031952190 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29e419af00 0x7f29e419b310 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:10.227 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 >> 172.21.15.115:0/4031952190 conn(0x7f29e40ff710 msgr2=0x7f29e41003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:10.227 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 shutdown_connections 2023-07-14T18:04:10.227 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:09.803+0000 7f29eaf6e700 1 -- 172.21.15.115:0/4031952190 wait complete. 2023-07-14T18:04:10.227 INFO:teuthology.orchestra.run.smithi115.stdout:Generating ssh key... 2023-07-14T18:04:11.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:10 smithi115 ceph-mon[127451]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:11.595 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: Generating public/private rsa key pair. 2023-07-14T18:04:11.595 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: Your identification has been saved in /tmp/tmpe137qki5/key. 2023-07-14T18:04:11.596 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: Your public key has been saved in /tmp/tmpe137qki5/key.pub. 2023-07-14T18:04:11.596 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: The key fingerprint is: 2023-07-14T18:04:11.596 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: SHA256:i2Ig1DexokLgYymWCucBqOakeyyJTkBOPE7m4iwKawY ceph-a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:04:11.596 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: The key's randomart image is: 2023-07-14T18:04:11.596 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: +---[RSA 3072]----+ 2023-07-14T18:04:11.596 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: |+ . | 2023-07-14T18:04:11.597 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: |=.+ o | 2023-07-14T18:04:11.597 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: |+^oo + | 2023-07-14T18:04:11.597 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: |^=+.o . | 2023-07-14T18:04:11.597 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: |@=o S | 2023-07-14T18:04:11.597 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: |Eo . . . | 2023-07-14T18:04:11.597 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: |=B o . . | 2023-07-14T18:04:11.597 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: |X+o. . | 2023-07-14T18:04:11.597 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: |*+ | 2023-07-14T18:04:11.598 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: +----[SHA256]-----+ 2023-07-14T18:04:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.684+0000 7f6d8e77a700 1 Processor -- start 2023-07-14T18:04:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.684+0000 7f6d8e77a700 1 -- start start 2023-07-14T18:04:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.684+0000 7f6d8e77a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d881049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.684+0000 7f6d8e77a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d88104ee0 con 0x7f6d88104590 2023-07-14T18:04:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.684+0000 7f6d87fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d881049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:11.664 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.684+0000 7f6d87fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d881049a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51370/0 (socket says 172.21.15.115:51370) 2023-07-14T18:04:11.665 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.684+0000 7f6d87fff700 1 -- 172.21.15.115:0/304034713 learned_addr learned my addr 172.21.15.115:0/304034713 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:11.665 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d87fff700 1 -- 172.21.15.115:0/304034713 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d88105020 con 0x7f6d88104590 2023-07-14T18:04:11.665 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d87fff700 1 --2- 172.21.15.115:0/304034713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d881049a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6d780098a0 tx=0x7f6d78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4c3f621e7d2a62 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:11.665 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d86ffd700 1 -- 172.21.15.115:0/304034713 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d78004160 con 0x7f6d88104590 2023-07-14T18:04:11.665 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d86ffd700 1 -- 172.21.15.115:0/304034713 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6d780042c0 con 0x7f6d88104590 2023-07-14T18:04:11.665 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d86ffd700 1 -- 172.21.15.115:0/304034713 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d78011940 con 0x7f6d88104590 2023-07-14T18:04:11.665 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d8e77a700 1 -- 172.21.15.115:0/304034713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 msgr2=0x7f6d881049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:11.666 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d8e77a700 1 --2- 172.21.15.115:0/304034713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d881049a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6d780098a0 tx=0x7f6d78009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:11.666 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d8e77a700 1 -- 172.21.15.115:0/304034713 shutdown_connections 2023-07-14T18:04:11.666 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d8e77a700 1 --2- 172.21.15.115:0/304034713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d881049a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:11.666 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.685+0000 7f6d8e77a700 1 -- 172.21.15.115:0/304034713 >> 172.21.15.115:0/304034713 conn(0x7f6d880fee80 msgr2=0x7f6d88101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:11.666 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.686+0000 7f6d8e77a700 1 -- 172.21.15.115:0/304034713 shutdown_connections 2023-07-14T18:04:11.666 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.686+0000 7f6d8e77a700 1 -- 172.21.15.115:0/304034713 wait complete. 2023-07-14T18:04:11.666 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.686+0000 7f6d8e77a700 1 Processor -- start 2023-07-14T18:04:11.666 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.686+0000 7f6d8e77a700 1 -- start start 2023-07-14T18:04:11.667 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d8e77a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d88103d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:11.667 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d8e77a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8819ba50 con 0x7f6d88104590 2023-07-14T18:04:11.667 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d87fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d88103d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:11.667 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d87fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d88103d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51372/0 (socket says 172.21.15.115:51372) 2023-07-14T18:04:11.667 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d87fff700 1 -- 172.21.15.115:0/1568397106 learned_addr learned my addr 172.21.15.115:0/1568397106 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:11.667 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d87fff700 1 -- 172.21.15.115:0/1568397106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78009550 con 0x7f6d88104590 2023-07-14T18:04:11.667 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d87fff700 1 --2- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d88103d50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d78000c00 tx=0x7f6d78011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:11.667 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d857fa700 1 -- 172.21.15.115:0/1568397106 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d780036a0 con 0x7f6d88104590 2023-07-14T18:04:11.668 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d857fa700 1 -- 172.21.15.115:0/1568397106 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6d7801b440 con 0x7f6d88104590 2023-07-14T18:04:11.668 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8819bb90 con 0x7f6d88104590 2023-07-14T18:04:11.668 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.687+0000 7f6d857fa700 1 -- 172.21.15.115:0/1568397106 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d780193f0 con 0x7f6d88104590 2023-07-14T18:04:11.668 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.688+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8819bfb0 con 0x7f6d88104590 2023-07-14T18:04:11.668 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.688+0000 7f6d857fa700 1 -- 172.21.15.115:0/1568397106 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f6d78019630 con 0x7f6d88104590 2023-07-14T18:04:11.668 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.688+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d88061c70 con 0x7f6d88104590 2023-07-14T18:04:11.668 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.689+0000 7f6d857fa700 1 --2- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f6d740387d0 0x7f6d7403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:11.669 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.689+0000 7f6d857fa700 1 -- 172.21.15.115:0/1568397106 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6d7804c7f0 con 0x7f6d88104590 2023-07-14T18:04:11.669 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.689+0000 7f6d877fe700 1 --2- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f6d740387d0 0x7f6d7403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:11.669 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.689+0000 7f6d877fe700 1 --2- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f6d740387d0 0x7f6d7403ac80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6d70006fd0 tx=0x7f6d70006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:11.669 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.693+0000 7f6d857fa700 1 -- 172.21.15.115:0/1568397106 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6d78018ce0 con 0x7f6d88104590 2023-07-14T18:04:11.669 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:10.878+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d88100220 con 0x7f6d740387d0 2023-07-14T18:04:11.669 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.232+0000 7f6d857fa700 1 -- 172.21.15.115:0/1568397106 <== mgr.14118 v2:172.21.15.115:6800/2169355495 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6d88100220 con 0x7f6d740387d0 2023-07-14T18:04:11.669 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.234+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f6d740387d0 msgr2=0x7f6d7403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:11.669 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.234+0000 7f6d8e77a700 1 --2- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f6d740387d0 0x7f6d7403ac80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6d70006fd0 tx=0x7f6d70006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:11.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.234+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 msgr2=0x7f6d88103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:11.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.234+0000 7f6d8e77a700 1 --2- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d88103d50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6d78000c00 tx=0x7f6d78011e70 comp rx=0 tx=0).stop 2023-07-14T18:04:11.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.234+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 shutdown_connections 2023-07-14T18:04:11.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.234+0000 7f6d8e77a700 1 --2- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f6d740387d0 0x7f6d7403ac80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:11.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.234+0000 7f6d8e77a700 1 --2- 172.21.15.115:0/1568397106 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d88104590 0x7f6d88103d50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:11.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.234+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 >> 172.21.15.115:0/1568397106 conn(0x7f6d880fee80 msgr2=0x7f6d880ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:11.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.235+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 shutdown_connections 2023-07-14T18:04:11.670 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:11.235+0000 7f6d8e77a700 1 -- 172.21.15.115:0/1568397106 wait complete. 2023-07-14T18:04:11.902 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-mon[127451]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:11.902 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-mon[127451]: Generating ssh key... 2023-07-14T18:04:11.902 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:11.902 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:11 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:12.704 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDg98lIYrnbK9S3Fx7eGxRMbSstfdjll8LuQQa0sN56n/yY5YbAKJUUbsCDX59Jhp3ZZcoPhoS1DI2qExikA6bk5Km46qjGRAACeLPpI1KrO+FdLeFBPcCIVBjfXXVsD8u0vSJUmvze+9ULrMWDdmq31msr3qqqTEv9kkMCBYXjEEr3WbhQwV8S18T0IhYfEAugAcvegjm1+PCVhTo1aNRVLU+uxSFoMsCrLki7EvW3xRBlTL+/r+6fiGrXvCxcFjHp6cgcBuQPzFfU/6VmVIzmECq8eWv1EYYT9Uu9Da7az/LnmPnD4r2ypjDFobJhJ8yA9D3Qd9Cuh/1I2B/G+rIxkYIhg6Lc2oFlBoYT+rxd+aJbehVUJtSFIkz1fPh6G5r6A/Hyj6Acx3UMe9EXxHpOX2r3Mf8A8HPEzAoD+qjn7nB23r8ovT3taDqJBigxSlLP1im1AWInM20VRe9vTkOyqnXxgX58SfFIYloi+9fXjWmcP0OdtDB10Cj6teOjkC0= ceph-a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:04:12.705 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.126+0000 7f21dccf4700 1 Processor -- start 2023-07-14T18:04:12.705 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.126+0000 7f21dccf4700 1 -- start start 2023-07-14T18:04:12.705 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.126+0000 7f21dccf4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d8079cd0 0x7f21d810a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:12.705 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.126+0000 7f21dccf4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8105b10 con 0x7f21d8079cd0 2023-07-14T18:04:12.705 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.126+0000 7f21d659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d8079cd0 0x7f21d810a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:12.705 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.126+0000 7f21d659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d8079cd0 0x7f21d810a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51376/0 (socket says 172.21.15.115:51376) 2023-07-14T18:04:12.705 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.126+0000 7f21d659c700 1 -- 172.21.15.115:0/2628539926 learned_addr learned my addr 172.21.15.115:0/2628539926 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:12.706 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.126+0000 7f21d659c700 1 -- 172.21.15.115:0/2628539926 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d810a660 con 0x7f21d8079cd0 2023-07-14T18:04:12.706 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21d659c700 1 --2- 172.21.15.115:0/2628539926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d8079cd0 0x7f21d810a120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f21c80098a0 tx=0x7f21c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d9763b8c5ffd2f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:12.706 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21d559a700 1 -- 172.21.15.115:0/2628539926 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21c8004160 con 0x7f21d8079cd0 2023-07-14T18:04:12.706 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21d559a700 1 -- 172.21.15.115:0/2628539926 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f21c80042c0 con 0x7f21d8079cd0 2023-07-14T18:04:12.706 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21d559a700 1 -- 172.21.15.115:0/2628539926 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21c8011940 con 0x7f21d8079cd0 2023-07-14T18:04:12.706 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21dccf4700 1 -- 172.21.15.115:0/2628539926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d8079cd0 msgr2=0x7f21d810a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:12.706 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21dccf4700 1 --2- 172.21.15.115:0/2628539926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d8079cd0 0x7f21d810a120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f21c80098a0 tx=0x7f21c8009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:12.707 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21dccf4700 1 -- 172.21.15.115:0/2628539926 shutdown_connections 2023-07-14T18:04:12.707 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21dccf4700 1 --2- 172.21.15.115:0/2628539926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d8079cd0 0x7f21d810a120 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:12.707 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21dccf4700 1 -- 172.21.15.115:0/2628539926 >> 172.21.15.115:0/2628539926 conn(0x7f21d80ff710 msgr2=0x7f21d8101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:12.707 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21dccf4700 1 -- 172.21.15.115:0/2628539926 shutdown_connections 2023-07-14T18:04:12.707 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.127+0000 7f21dccf4700 1 -- 172.21.15.115:0/2628539926 wait complete. 2023-07-14T18:04:12.707 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.128+0000 7f21dccf4700 1 Processor -- start 2023-07-14T18:04:12.707 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.128+0000 7f21dccf4700 1 -- start start 2023-07-14T18:04:12.707 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21dccf4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d819af00 0x7f21d819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:12.708 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21dccf4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d819b850 con 0x7f21d819af00 2023-07-14T18:04:12.708 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21d659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d819af00 0x7f21d819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:12.708 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21d659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d819af00 0x7f21d819b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51378/0 (socket says 172.21.15.115:51378) 2023-07-14T18:04:12.708 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21d659c700 1 -- 172.21.15.115:0/3431794064 learned_addr learned my addr 172.21.15.115:0/3431794064 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:12.708 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21d659c700 1 -- 172.21.15.115:0/3431794064 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c8009550 con 0x7f21d819af00 2023-07-14T18:04:12.708 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21d659c700 1 --2- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d819af00 0x7f21d819b310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f21c8006af0 tx=0x7f21c8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:12.708 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21c77fe700 1 -- 172.21.15.115:0/3431794064 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21c8003bf0 con 0x7f21d819af00 2023-07-14T18:04:12.708 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21c77fe700 1 -- 172.21.15.115:0/3431794064 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f21c801b8b0 con 0x7f21d819af00 2023-07-14T18:04:12.709 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d819ba50 con 0x7f21d819af00 2023-07-14T18:04:12.709 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21c77fe700 1 -- 172.21.15.115:0/3431794064 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21c80198b0 con 0x7f21d819af00 2023-07-14T18:04:12.709 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.129+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d819e6b0 con 0x7f21d819af00 2023-07-14T18:04:12.709 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.130+0000 7f21c77fe700 1 -- 172.21.15.115:0/3431794064 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f21c8019b20 con 0x7f21d819af00 2023-07-14T18:04:12.709 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.130+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d8079cd0 con 0x7f21d819af00 2023-07-14T18:04:12.709 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.130+0000 7f21c77fe700 1 --2- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f21c00387d0 0x7f21c003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:12.709 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.130+0000 7f21c77fe700 1 -- 172.21.15.115:0/3431794064 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f21c804ca00 con 0x7f21d819af00 2023-07-14T18:04:12.710 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.130+0000 7f21d5d9b700 1 --2- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f21c00387d0 0x7f21c003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:12.710 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.131+0000 7f21d5d9b700 1 --2- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f21c00387d0 0x7f21c003ac80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f21cc006fd0 tx=0x7f21cc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:12.710 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.135+0000 7f21c77fe700 1 -- 172.21.15.115:0/3431794064 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f21c8018db0 con 0x7f21d819af00 2023-07-14T18:04:12.710 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.320+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f21d8100b00 con 0x7f21c00387d0 2023-07-14T18:04:12.710 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.321+0000 7f21c77fe700 1 -- 172.21.15.115:0/3431794064 <== mgr.14118 v2:172.21.15.115:6800/2169355495 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f21d8100b00 con 0x7f21c00387d0 2023-07-14T18:04:12.710 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f21c00387d0 msgr2=0x7f21c003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:12.710 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 --2- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f21c00387d0 0x7f21c003ac80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f21cc006fd0 tx=0x7f21cc006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:12.710 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d819af00 msgr2=0x7f21d819b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:12.711 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 --2- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d819af00 0x7f21d819b310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f21c8006af0 tx=0x7f21c8011a30 comp rx=0 tx=0).stop 2023-07-14T18:04:12.711 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 shutdown_connections 2023-07-14T18:04:12.711 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 --2- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f21c00387d0 0x7f21c003ac80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:12.711 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 --2- 172.21.15.115:0/3431794064 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f21d819af00 0x7f21d819b310 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:12.711 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 >> 172.21.15.115:0/3431794064 conn(0x7f21d80ff710 msgr2=0x7f21d81003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:12.711 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 shutdown_connections 2023-07-14T18:04:12.711 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:12.323+0000 7f21dccf4700 1 -- 172.21.15.115:0/3431794064 wait complete. 2023-07-14T18:04:12.711 INFO:teuthology.orchestra.run.smithi115.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-14T18:04:12.712 INFO:teuthology.orchestra.run.smithi115.stdout:Adding key to root@localhost authorized_keys... 2023-07-14T18:04:12.712 INFO:teuthology.orchestra.run.smithi115.stdout:Adding host smithi115... 2023-07-14T18:04:13.614 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:13 smithi115 ceph-mon[127451]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:14.595 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:14 smithi115 ceph-mon[127451]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "addr": "172.21.15.115", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:14.596 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:14 smithi115 ceph-mon[127451]: Deploying cephadm binary to smithi115 2023-07-14T18:04:16.061 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout Added host 'smithi115' with addr '172.21.15.115' 2023-07-14T18:04:16.061 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.133+0000 7f2ad288d700 1 Processor -- start 2023-07-14T18:04:16.062 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.133+0000 7f2ad288d700 1 -- start start 2023-07-14T18:04:16.062 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.133+0000 7f2ad288d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:16.062 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.133+0000 7f2ad288d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc1070e0 con 0x7f2acc106790 2023-07-14T18:04:16.062 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.133+0000 7f2acbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:16.063 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.133+0000 7f2acbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc106ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51382/0 (socket says 172.21.15.115:51382) 2023-07-14T18:04:16.063 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.133+0000 7f2acbfff700 1 -- 172.21.15.115:0/1283286160 learned_addr learned my addr 172.21.15.115:0/1283286160 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:16.063 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.134+0000 7f2acbfff700 1 -- 172.21.15.115:0/1283286160 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc107220 con 0x7f2acc106790 2023-07-14T18:04:16.064 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.134+0000 7f2acbfff700 1 --2- 172.21.15.115:0/1283286160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc106ba0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40098a0 tx=0x7f2ab4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa55557de76cec55 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:16.064 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.134+0000 7f2acaffd700 1 -- 172.21.15.115:0/1283286160 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ab4004160 con 0x7f2acc106790 2023-07-14T18:04:16.064 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.134+0000 7f2acaffd700 1 -- 172.21.15.115:0/1283286160 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2ab40042c0 con 0x7f2acc106790 2023-07-14T18:04:16.065 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.134+0000 7f2acaffd700 1 -- 172.21.15.115:0/1283286160 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ab4011940 con 0x7f2acc106790 2023-07-14T18:04:16.065 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.134+0000 7f2ad288d700 1 -- 172.21.15.115:0/1283286160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 msgr2=0x7f2acc106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:16.065 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.134+0000 7f2ad288d700 1 --2- 172.21.15.115:0/1283286160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc106ba0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40098a0 tx=0x7f2ab4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:16.065 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.135+0000 7f2ad288d700 1 -- 172.21.15.115:0/1283286160 shutdown_connections 2023-07-14T18:04:16.066 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.135+0000 7f2ad288d700 1 --2- 172.21.15.115:0/1283286160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc106ba0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:16.066 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.135+0000 7f2ad288d700 1 -- 172.21.15.115:0/1283286160 >> 172.21.15.115:0/1283286160 conn(0x7f2acc0fee80 msgr2=0x7f2acc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:16.066 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.135+0000 7f2ad288d700 1 -- 172.21.15.115:0/1283286160 shutdown_connections 2023-07-14T18:04:16.067 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.135+0000 7f2ad288d700 1 -- 172.21.15.115:0/1283286160 wait complete. 2023-07-14T18:04:16.067 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.135+0000 7f2ad288d700 1 Processor -- start 2023-07-14T18:04:16.067 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.135+0000 7f2ad288d700 1 -- start start 2023-07-14T18:04:16.067 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.135+0000 7f2ad288d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:16.068 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.135+0000 7f2ad288d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc1991b0 con 0x7f2acc106790 2023-07-14T18:04:16.068 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2acbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:16.068 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2acbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51384/0 (socket says 172.21.15.115:51384) 2023-07-14T18:04:16.069 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2acbfff700 1 -- 172.21.15.115:0/1325427662 learned_addr learned my addr 172.21.15.115:0/1325427662 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:16.069 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2acbfff700 1 -- 172.21.15.115:0/1325427662 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4009550 con 0x7f2acc106790 2023-07-14T18:04:16.069 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2acbfff700 1 --2- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc198c70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4011dc0 tx=0x7f2ab4011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:16.070 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2ac97fa700 1 -- 172.21.15.115:0/1325427662 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ab40037d0 con 0x7f2acc106790 2023-07-14T18:04:16.070 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2ac97fa700 1 -- 172.21.15.115:0/1325427662 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2ab401b440 con 0x7f2acc106790 2023-07-14T18:04:16.070 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2ac97fa700 1 -- 172.21.15.115:0/1325427662 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ab401a560 con 0x7f2acc106790 2023-07-14T18:04:16.070 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc1993b0 con 0x7f2acc106790 2023-07-14T18:04:16.071 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.136+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc1997d0 con 0x7f2acc106790 2023-07-14T18:04:16.071 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.137+0000 7f2ac97fa700 1 -- 172.21.15.115:0/1325427662 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f2ab401a820 con 0x7f2acc106790 2023-07-14T18:04:16.071 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.137+0000 7f2ac97fa700 1 --2- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f2ab80387d0 0x7f2ab803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:16.072 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.137+0000 7f2ac97fa700 1 -- 172.21.15.115:0/1325427662 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2ab404c660 con 0x7f2acc106790 2023-07-14T18:04:16.072 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.137+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc061c70 con 0x7f2acc106790 2023-07-14T18:04:16.072 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.137+0000 7f2acb7fe700 1 --2- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f2ab80387d0 0x7f2ab803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:16.073 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.138+0000 7f2acb7fe700 1 --2- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f2ab80387d0 0x7f2ab803ac80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2abc006fd0 tx=0x7f2abc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:16.073 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.142+0000 7f2ac97fa700 1 -- 172.21.15.115:0/1325427662 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2ab401f080 con 0x7f2acc106790 2023-07-14T18:04:16.073 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:13.332+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi115", "addr": "172.21.15.115", "target": ["mon-mgr", ""]}) v1 -- 0x7f2acc100190 con 0x7f2ab80387d0 2023-07-14T18:04:16.073 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.632+0000 7f2ac97fa700 1 -- 172.21.15.115:0/1325427662 <== mgr.14118 v2:172.21.15.115:6800/2169355495 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2acc100190 con 0x7f2ab80387d0 2023-07-14T18:04:16.074 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f2ab80387d0 msgr2=0x7f2ab803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:16.074 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 --2- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f2ab80387d0 0x7f2ab803ac80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2abc006fd0 tx=0x7f2abc006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:16.074 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 msgr2=0x7f2acc198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:16.075 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 --2- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc198c70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4011dc0 tx=0x7f2ab4011ea0 comp rx=0 tx=0).stop 2023-07-14T18:04:16.075 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 shutdown_connections 2023-07-14T18:04:16.075 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 --2- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f2ab80387d0 0x7f2ab803ac80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:16.076 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 --2- 172.21.15.115:0/1325427662 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2acc106790 0x7f2acc198c70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:16.076 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 >> 172.21.15.115:0/1325427662 conn(0x7f2acc0fee80 msgr2=0x7f2acc0ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:16.076 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 shutdown_connections 2023-07-14T18:04:16.076 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:15.635+0000 7f2ad288d700 1 -- 172.21.15.115:0/1325427662 wait complete. 2023-07-14T18:04:16.077 INFO:teuthology.orchestra.run.smithi115.stdout:Deploying unmanaged mon service... 2023-07-14T18:04:17.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:16 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:17.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:16 smithi115 ceph-mon[127451]: Added host smithi115 2023-07-14T18:04:17.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:16 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:17.202 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-14T18:04:17.203 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.499+0000 7fc6be325700 1 Processor -- start 2023-07-14T18:04:17.203 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.499+0000 7fc6be325700 1 -- start start 2023-07-14T18:04:17.203 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6be325700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b8079cd0 0x7fc6b810a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:17.203 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6be325700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8105b00 con 0x7fc6b8079cd0 2023-07-14T18:04:17.203 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6b7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b8079cd0 0x7fc6b810a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:17.203 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6b7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b8079cd0 0x7fc6b810a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51418/0 (socket says 172.21.15.115:51418) 2023-07-14T18:04:17.203 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6b7fff700 1 -- 172.21.15.115:0/3957900072 learned_addr learned my addr 172.21.15.115:0/3957900072 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:17.204 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6b7fff700 1 -- 172.21.15.115:0/3957900072 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b810a650 con 0x7fc6b8079cd0 2023-07-14T18:04:17.204 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6b7fff700 1 --2- 172.21.15.115:0/3957900072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b8079cd0 0x7fc6b810a110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80098a0 tx=0x7fc6a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a25c00e006ea69 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:17.204 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6b6ffd700 1 -- 172.21.15.115:0/3957900072 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6a8004160 con 0x7fc6b8079cd0 2023-07-14T18:04:17.204 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6b6ffd700 1 -- 172.21.15.115:0/3957900072 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc6a80042c0 con 0x7fc6b8079cd0 2023-07-14T18:04:17.204 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.500+0000 7fc6b6ffd700 1 -- 172.21.15.115:0/3957900072 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6a8011940 con 0x7fc6b8079cd0 2023-07-14T18:04:17.204 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.501+0000 7fc6be325700 1 -- 172.21.15.115:0/3957900072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b8079cd0 msgr2=0x7fc6b810a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:17.204 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.501+0000 7fc6be325700 1 --2- 172.21.15.115:0/3957900072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b8079cd0 0x7fc6b810a110 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80098a0 tx=0x7fc6a8009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:17.205 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.501+0000 7fc6be325700 1 -- 172.21.15.115:0/3957900072 shutdown_connections 2023-07-14T18:04:17.205 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.501+0000 7fc6be325700 1 --2- 172.21.15.115:0/3957900072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b8079cd0 0x7fc6b810a110 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:17.205 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.501+0000 7fc6be325700 1 -- 172.21.15.115:0/3957900072 >> 172.21.15.115:0/3957900072 conn(0x7fc6b80ff6e0 msgr2=0x7fc6b8101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:17.205 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.501+0000 7fc6be325700 1 -- 172.21.15.115:0/3957900072 shutdown_connections 2023-07-14T18:04:17.205 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.501+0000 7fc6be325700 1 -- 172.21.15.115:0/3957900072 wait complete. 2023-07-14T18:04:17.205 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.501+0000 7fc6be325700 1 Processor -- start 2023-07-14T18:04:17.205 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6be325700 1 -- start start 2023-07-14T18:04:17.205 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6be325700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b819ae80 0x7fc6b819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:17.206 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6be325700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b819b7d0 con 0x7fc6b819ae80 2023-07-14T18:04:17.206 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6b7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b819ae80 0x7fc6b819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:17.206 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6b7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b819ae80 0x7fc6b819b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51420/0 (socket says 172.21.15.115:51420) 2023-07-14T18:04:17.206 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6b7fff700 1 -- 172.21.15.115:0/2256420681 learned_addr learned my addr 172.21.15.115:0/2256420681 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:17.206 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6b7fff700 1 -- 172.21.15.115:0/2256420681 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a8009550 con 0x7fc6b819ae80 2023-07-14T18:04:17.206 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6b7fff700 1 --2- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b819ae80 0x7fc6b819b290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8000c00 tx=0x7fc6a8011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:17.206 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6b57fa700 1 -- 172.21.15.115:0/2256420681 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6a80036a0 con 0x7fc6b819ae80 2023-07-14T18:04:17.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6b57fa700 1 -- 172.21.15.115:0/2256420681 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc6a801b440 con 0x7fc6b819ae80 2023-07-14T18:04:17.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6b57fa700 1 -- 172.21.15.115:0/2256420681 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6a80193f0 con 0x7fc6b819ae80 2023-07-14T18:04:17.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.502+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b819b9d0 con 0x7fc6b819ae80 2023-07-14T18:04:17.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.503+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b819e630 con 0x7fc6b819ae80 2023-07-14T18:04:17.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.503+0000 7fc6b57fa700 1 -- 172.21.15.115:0/2256420681 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc6a801b5b0 con 0x7fc6b819ae80 2023-07-14T18:04:17.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.503+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b8079cd0 con 0x7fc6b819ae80 2023-07-14T18:04:17.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.504+0000 7fc6b57fa700 1 --2- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fc6a00387d0 0x7fc6a003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:17.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.504+0000 7fc6b57fa700 1 -- 172.21.15.115:0/2256420681 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc6a804c600 con 0x7fc6b819ae80 2023-07-14T18:04:17.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.504+0000 7fc6b77fe700 1 --2- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fc6a00387d0 0x7fc6a003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:17.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.504+0000 7fc6b77fe700 1 --2- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fc6a00387d0 0x7fc6a003ac80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac006fd0 tx=0x7fc6ac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:17.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.508+0000 7fc6b57fa700 1 -- 172.21.15.115:0/2256420681 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc6a8022950 con 0x7fc6b819ae80 2023-07-14T18:04:17.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.723+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc6b8100a20 con 0x7fc6a00387d0 2023-07-14T18:04:17.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.727+0000 7fc6b57fa700 1 -- 172.21.15.115:0/2256420681 <== mgr.14118 v2:172.21.15.115:6800/2169355495 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc6b8100a20 con 0x7fc6a00387d0 2023-07-14T18:04:17.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fc6a00387d0 msgr2=0x7fc6a003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:17.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 --2- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fc6a00387d0 0x7fc6a003ac80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac006fd0 tx=0x7fc6ac006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:17.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b819ae80 msgr2=0x7fc6b819b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:17.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 --2- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b819ae80 0x7fc6b819b290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8000c00 tx=0x7fc6a8011e70 comp rx=0 tx=0).stop 2023-07-14T18:04:17.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 shutdown_connections 2023-07-14T18:04:17.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 --2- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fc6a00387d0 0x7fc6a003ac80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:17.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 --2- 172.21.15.115:0/2256420681 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc6b819ae80 0x7fc6b819b290 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:17.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 >> 172.21.15.115:0/2256420681 conn(0x7fc6b80ff6e0 msgr2=0x7fc6b8100310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:17.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 shutdown_connections 2023-07-14T18:04:17.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:16.730+0000 7fc6be325700 1 -- 172.21.15.115:0/2256420681 wait complete. 2023-07-14T18:04:17.210 INFO:teuthology.orchestra.run.smithi115.stdout:Deploying unmanaged mgr service... 2023-07-14T18:04:17.845 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:17 smithi115 ceph-mon[127451]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:17.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:17 smithi115 ceph-mon[127451]: Saving service mon spec with placement count:5 2023-07-14T18:04:17.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:17 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:18.600 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-14T18:04:18.601 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.940+0000 7fe57e136700 1 Processor -- start 2023-07-14T18:04:18.601 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.940+0000 7fe57e136700 1 -- start start 2023-07-14T18:04:18.601 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.940+0000 7fe57e136700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:18.601 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.940+0000 7fe57e136700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe578104ec0 con 0x7fe578104570 2023-07-14T18:04:18.601 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.940+0000 7fe5777fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:18.601 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.940+0000 7fe5777fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51424/0 (socket says 172.21.15.115:51424) 2023-07-14T18:04:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.940+0000 7fe5777fe700 1 -- 172.21.15.115:0/3603645638 learned_addr learned my addr 172.21.15.115:0/3603645638 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.941+0000 7fe5777fe700 1 -- 172.21.15.115:0/3603645638 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe578105000 con 0x7fe578104570 2023-07-14T18:04:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.941+0000 7fe5777fe700 1 --2- 172.21.15.115:0/3603645638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578104980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0098a0 tx=0x7fe56c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59aa1b3af71c72a0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.941+0000 7fe5767fc700 1 -- 172.21.15.115:0/3603645638 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe56c004160 con 0x7fe578104570 2023-07-14T18:04:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.941+0000 7fe5767fc700 1 -- 172.21.15.115:0/3603645638 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe56c0042c0 con 0x7fe578104570 2023-07-14T18:04:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.941+0000 7fe5767fc700 1 -- 172.21.15.115:0/3603645638 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe56c011940 con 0x7fe578104570 2023-07-14T18:04:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.941+0000 7fe57e136700 1 -- 172.21.15.115:0/3603645638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 msgr2=0x7fe578104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:18.602 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.941+0000 7fe57e136700 1 --2- 172.21.15.115:0/3603645638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578104980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0098a0 tx=0x7fe56c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:18.603 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.942+0000 7fe57e136700 1 -- 172.21.15.115:0/3603645638 shutdown_connections 2023-07-14T18:04:18.603 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.942+0000 7fe57e136700 1 --2- 172.21.15.115:0/3603645638 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578104980 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:18.603 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.942+0000 7fe57e136700 1 -- 172.21.15.115:0/3603645638 >> 172.21.15.115:0/3603645638 conn(0x7fe5780fee80 msgr2=0x7fe578101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:18.603 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.942+0000 7fe57e136700 1 -- 172.21.15.115:0/3603645638 shutdown_connections 2023-07-14T18:04:18.603 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.942+0000 7fe57e136700 1 -- 172.21.15.115:0/3603645638 wait complete. 2023-07-14T18:04:18.603 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.942+0000 7fe57e136700 1 Processor -- start 2023-07-14T18:04:18.603 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe57e136700 1 -- start start 2023-07-14T18:04:18.603 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe57e136700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578103cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:18.604 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe57e136700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57819b9e0 con 0x7fe578104570 2023-07-14T18:04:18.604 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe5777fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578103cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:18.604 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe5777fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578103cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51426/0 (socket says 172.21.15.115:51426) 2023-07-14T18:04:18.604 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe5777fe700 1 -- 172.21.15.115:0/2337490479 learned_addr learned my addr 172.21.15.115:0/2337490479 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:18.604 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe5777fe700 1 -- 172.21.15.115:0/2337490479 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c009550 con 0x7fe578104570 2023-07-14T18:04:18.604 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe5777fe700 1 --2- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578103cf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe56c000c00 tx=0x7fe56c011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:18.604 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe574ff9700 1 -- 172.21.15.115:0/2337490479 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe56c0036a0 con 0x7fe578104570 2023-07-14T18:04:18.605 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe574ff9700 1 -- 172.21.15.115:0/2337490479 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe56c01b440 con 0x7fe578104570 2023-07-14T18:04:18.605 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe57e136700 1 -- 172.21.15.115:0/2337490479 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57819bb20 con 0x7fe578104570 2023-07-14T18:04:18.605 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.943+0000 7fe574ff9700 1 -- 172.21.15.115:0/2337490479 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe56c0193f0 con 0x7fe578104570 2023-07-14T18:04:18.605 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.944+0000 7fe57e136700 1 -- 172.21.15.115:0/2337490479 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57819bf40 con 0x7fe578104570 2023-07-14T18:04:18.605 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.944+0000 7fe574ff9700 1 -- 172.21.15.115:0/2337490479 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe56c0195e0 con 0x7fe578104570 2023-07-14T18:04:18.605 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.944+0000 7fe55e7fc700 1 -- 172.21.15.115:0/2337490479 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5580052f0 con 0x7fe578104570 2023-07-14T18:04:18.605 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.945+0000 7fe574ff9700 1 --2- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fe560038820 0x7fe56003acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:18.605 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.945+0000 7fe574ff9700 1 -- 172.21.15.115:0/2337490479 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe56c04c840 con 0x7fe578104570 2023-07-14T18:04:18.606 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.945+0000 7fe576ffd700 1 --2- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fe560038820 0x7fe56003acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:18.606 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.945+0000 7fe576ffd700 1 --2- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fe560038820 0x7fe56003acd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe568006fd0 tx=0x7fe568006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:18.606 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:17.949+0000 7fe574ff9700 1 -- 172.21.15.115:0/2337490479 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe56c018ca0 con 0x7fe578104570 2023-07-14T18:04:18.606 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.003+0000 7fe574ff9700 1 -- 172.21.15.115:0/2337490479 <== mon.0 v2:172.21.15.115:3300/0 7 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe56c04c2b0 con 0x7fe578104570 2023-07-14T18:04:18.606 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.162+0000 7fe55e7fc700 1 -- 172.21.15.115:0/2337490479 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe558000bc0 con 0x7fe560038820 2023-07-14T18:04:18.606 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.169+0000 7fe574ff9700 1 -- 172.21.15.115:0/2337490479 <== mgr.14118 v2:172.21.15.115:6800/2169355495 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe558000bc0 con 0x7fe560038820 2023-07-14T18:04:18.606 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.171+0000 7fe55e7fc700 1 -- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fe560038820 msgr2=0x7fe56003acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:18.606 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.171+0000 7fe55e7fc700 1 --2- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fe560038820 0x7fe56003acd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe568006fd0 tx=0x7fe568006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:18.607 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.171+0000 7fe55e7fc700 1 -- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 msgr2=0x7fe578103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:18.607 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.171+0000 7fe55e7fc700 1 --2- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578103cf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe56c000c00 tx=0x7fe56c011e70 comp rx=0 tx=0).stop 2023-07-14T18:04:18.607 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.171+0000 7fe55e7fc700 1 -- 172.21.15.115:0/2337490479 shutdown_connections 2023-07-14T18:04:18.607 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.171+0000 7fe55e7fc700 1 --2- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7fe560038820 0x7fe56003acd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:18.607 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.171+0000 7fe55e7fc700 1 --2- 172.21.15.115:0/2337490479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe578104570 0x7fe578103cf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:18.607 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.171+0000 7fe55e7fc700 1 -- 172.21.15.115:0/2337490479 >> 172.21.15.115:0/2337490479 conn(0x7fe5780fee80 msgr2=0x7fe5780ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:18.607 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.171+0000 7fe55e7fc700 1 -- 172.21.15.115:0/2337490479 shutdown_connections 2023-07-14T18:04:18.608 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:18.172+0000 7fe55e7fc700 1 -- 172.21.15.115:0/2337490479 wait complete. 2023-07-14T18:04:19.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:19 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:19.095 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:19 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:19 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:19.643 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f1acba700 1 Processor -- start 2023-07-14T18:04:19.643 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f1acba700 1 -- start start 2023-07-14T18:04:19.643 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f1acba700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f14103890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:19.643 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f1acba700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f14103dd0 con 0x7f1f14103480 2023-07-14T18:04:19.643 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f18a56700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f14103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:19.643 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f18a56700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f14103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51430/0 (socket says 172.21.15.115:51430) 2023-07-14T18:04:19.644 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f18a56700 1 -- 172.21.15.115:0/2114464423 learned_addr learned my addr 172.21.15.115:0/2114464423 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:19.644 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f18a56700 1 -- 172.21.15.115:0/2114464423 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f14103f10 con 0x7f1f14103480 2023-07-14T18:04:19.644 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f18a56700 1 --2- 172.21.15.115:0/2114464423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f14103890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f080098a0 tx=0x7f1f08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5024c442fb105d2c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:19.644 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f137fe700 1 -- 172.21.15.115:0/2114464423 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f08004160 con 0x7f1f14103480 2023-07-14T18:04:19.644 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.072+0000 7f1f137fe700 1 -- 172.21.15.115:0/2114464423 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1f080042c0 con 0x7f1f14103480 2023-07-14T18:04:19.644 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.073+0000 7f1f137fe700 1 -- 172.21.15.115:0/2114464423 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f080118e0 con 0x7f1f14103480 2023-07-14T18:04:19.644 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.073+0000 7f1f1acba700 1 -- 172.21.15.115:0/2114464423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 msgr2=0x7f1f14103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:19.644 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.073+0000 7f1f1acba700 1 --2- 172.21.15.115:0/2114464423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f14103890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f080098a0 tx=0x7f1f08009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:19.645 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.073+0000 7f1f1acba700 1 -- 172.21.15.115:0/2114464423 shutdown_connections 2023-07-14T18:04:19.645 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.073+0000 7f1f1acba700 1 --2- 172.21.15.115:0/2114464423 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f14103890 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:19.645 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.073+0000 7f1f1acba700 1 -- 172.21.15.115:0/2114464423 >> 172.21.15.115:0/2114464423 conn(0x7f1f140fee80 msgr2=0x7f1f14101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:19.645 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.073+0000 7f1f1acba700 1 -- 172.21.15.115:0/2114464423 shutdown_connections 2023-07-14T18:04:19.645 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.073+0000 7f1f1acba700 1 -- 172.21.15.115:0/2114464423 wait complete. 2023-07-14T18:04:19.645 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f1acba700 1 Processor -- start 2023-07-14T18:04:19.645 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f1acba700 1 -- start start 2023-07-14T18:04:19.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f1acba700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f1419f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:19.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f1acba700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1419f940 con 0x7f1f14103480 2023-07-14T18:04:19.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f18a56700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f1419f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:19.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f18a56700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f1419f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51432/0 (socket says 172.21.15.115:51432) 2023-07-14T18:04:19.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f18a56700 1 -- 172.21.15.115:0/2741843114 learned_addr learned my addr 172.21.15.115:0/2741843114 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:19.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f18a56700 1 -- 172.21.15.115:0/2741843114 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f08009550 con 0x7f1f14103480 2023-07-14T18:04:19.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f18a56700 1 --2- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f1419f400 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f08000c00 tx=0x7f1f080119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:19.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.074+0000 7f1f11ffb700 1 -- 172.21.15.115:0/2741843114 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f080036a0 con 0x7f1f14103480 2023-07-14T18:04:19.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.075+0000 7f1f11ffb700 1 -- 172.21.15.115:0/2741843114 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1f0801b440 con 0x7f1f14103480 2023-07-14T18:04:19.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.075+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1419fb40 con 0x7f1f14103480 2023-07-14T18:04:19.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.075+0000 7f1f11ffb700 1 -- 172.21.15.115:0/2741843114 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f0801a3d0 con 0x7f1f14103480 2023-07-14T18:04:19.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.075+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1419ff60 con 0x7f1f14103480 2023-07-14T18:04:19.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.076+0000 7f1f11ffb700 1 -- 172.21.15.115:0/2741843114 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f1f0801a690 con 0x7f1f14103480 2023-07-14T18:04:19.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.076+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f14061c70 con 0x7f1f14103480 2023-07-14T18:04:19.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.076+0000 7f1f11ffb700 1 --2- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f1f040387d0 0x7f1f0403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:19.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.076+0000 7f1f11ffb700 1 -- 172.21.15.115:0/2741843114 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1f0804c7c0 con 0x7f1f14103480 2023-07-14T18:04:19.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.076+0000 7f1f13fff700 1 --2- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f1f040387d0 0x7f1f0403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:19.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.076+0000 7f1f13fff700 1 --2- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f1f040387d0 0x7f1f0403ac80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f00006fd0 tx=0x7f1f00006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:19.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.080+0000 7f1f11ffb700 1 -- 172.21.15.115:0/2741843114 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1f08018d20 con 0x7f1f14103480 2023-07-14T18:04:19.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.283+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1f141a2fd0 con 0x7f1f14103480 2023-07-14T18:04:19.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.289+0000 7f1f11ffb700 1 -- 172.21.15.115:0/2741843114 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1f08018b40 con 0x7f1f14103480 2023-07-14T18:04:19.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f1f040387d0 msgr2=0x7f1f0403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:19.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 --2- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f1f040387d0 0x7f1f0403ac80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1f00006fd0 tx=0x7f1f00006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:19.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 msgr2=0x7f1f1419f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:19.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 --2- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f1419f400 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1f08000c00 tx=0x7f1f080119a0 comp rx=0 tx=0).stop 2023-07-14T18:04:19.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 shutdown_connections 2023-07-14T18:04:19.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 --2- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f1f040387d0 0x7f1f0403ac80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:19.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 --2- 172.21.15.115:0/2741843114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f14103480 0x7f1f1419f400 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:19.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 >> 172.21.15.115:0/2741843114 conn(0x7f1f140fee80 msgr2=0x7f1f14109960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:19.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 shutdown_connections 2023-07-14T18:04:19.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:19.291+0000 7f1f1acba700 1 -- 172.21.15.115:0/2741843114 wait complete. 2023-07-14T18:04:20.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:20 smithi115 ceph-mon[127451]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:20.345 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:20 smithi115 ceph-mon[127451]: Saving service mgr spec with placement count:2 2023-07-14T18:04:20.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:20 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2741843114' entity='client.admin' 2023-07-14T18:04:20.941 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.297+0000 7f9c44fa1700 1 Processor -- start 2023-07-14T18:04:20.941 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.298+0000 7f9c44fa1700 1 -- start start 2023-07-14T18:04:20.941 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.298+0000 7f9c44fa1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c40104320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:20.942 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.298+0000 7f9c44fa1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c40104860 con 0x7f9c40103ef0 2023-07-14T18:04:20.942 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.298+0000 7f9c3f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c40104320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:20.942 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.298+0000 7f9c3f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c40104320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51436/0 (socket says 172.21.15.115:51436) 2023-07-14T18:04:20.942 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.298+0000 7f9c3f7fe700 1 -- 172.21.15.115:0/4284926239 learned_addr learned my addr 172.21.15.115:0/4284926239 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:20.942 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.298+0000 7f9c3f7fe700 1 -- 172.21.15.115:0/4284926239 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c401049a0 con 0x7f9c40103ef0 2023-07-14T18:04:20.942 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.299+0000 7f9c3f7fe700 1 --2- 172.21.15.115:0/4284926239 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c40104320 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c300098a0 tx=0x7f9c30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddafb101fc60a476 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:20.942 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.299+0000 7f9c3e7fc700 1 -- 172.21.15.115:0/4284926239 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c30004160 con 0x7f9c40103ef0 2023-07-14T18:04:20.943 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.299+0000 7f9c3e7fc700 1 -- 172.21.15.115:0/4284926239 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9c300042c0 con 0x7f9c40103ef0 2023-07-14T18:04:20.943 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.299+0000 7f9c3e7fc700 1 -- 172.21.15.115:0/4284926239 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c300118e0 con 0x7f9c40103ef0 2023-07-14T18:04:20.943 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.299+0000 7f9c44fa1700 1 -- 172.21.15.115:0/4284926239 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 msgr2=0x7f9c40104320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:20.943 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.299+0000 7f9c44fa1700 1 --2- 172.21.15.115:0/4284926239 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c40104320 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c300098a0 tx=0x7f9c30009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:20.943 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.299+0000 7f9c44fa1700 1 -- 172.21.15.115:0/4284926239 shutdown_connections 2023-07-14T18:04:20.944 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.299+0000 7f9c44fa1700 1 --2- 172.21.15.115:0/4284926239 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c40104320 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:20.944 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.299+0000 7f9c44fa1700 1 -- 172.21.15.115:0/4284926239 >> 172.21.15.115:0/4284926239 conn(0x7f9c400ff4f0 msgr2=0x7f9c40101920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:20.944 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.300+0000 7f9c44fa1700 1 -- 172.21.15.115:0/4284926239 shutdown_connections 2023-07-14T18:04:20.944 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.300+0000 7f9c44fa1700 1 -- 172.21.15.115:0/4284926239 wait complete. 2023-07-14T18:04:20.944 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.300+0000 7f9c44fa1700 1 Processor -- start 2023-07-14T18:04:20.945 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c44fa1700 1 -- start start 2023-07-14T18:04:20.945 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c44fa1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c4018ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:20.945 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c44fa1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4018f4d0 con 0x7f9c40103ef0 2023-07-14T18:04:20.945 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c3f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c4018ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:20.945 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c3f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c4018ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51438/0 (socket says 172.21.15.115:51438) 2023-07-14T18:04:20.945 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c3f7fe700 1 -- 172.21.15.115:0/717867458 learned_addr learned my addr 172.21.15.115:0/717867458 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:20.945 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c3f7fe700 1 -- 172.21.15.115:0/717867458 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c30009550 con 0x7f9c40103ef0 2023-07-14T18:04:20.945 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c3f7fe700 1 --2- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c4018ef90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9c30006af0 tx=0x7f9c30011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:20.946 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c3cff9700 1 -- 172.21.15.115:0/717867458 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c300037d0 con 0x7f9c40103ef0 2023-07-14T18:04:20.946 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c3cff9700 1 -- 172.21.15.115:0/717867458 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9c3001b440 con 0x7f9c40103ef0 2023-07-14T18:04:20.946 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4018d640 con 0x7f9c40103ef0 2023-07-14T18:04:20.946 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.301+0000 7f9c3cff9700 1 -- 172.21.15.115:0/717867458 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c30019560 con 0x7f9c40103ef0 2023-07-14T18:04:20.946 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.302+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4018dac0 con 0x7f9c40103ef0 2023-07-14T18:04:20.946 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.302+0000 7f9c3cff9700 1 -- 172.21.15.115:0/717867458 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f9c30019820 con 0x7f9c40103ef0 2023-07-14T18:04:20.946 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.302+0000 7f9c3cff9700 1 --2- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f9c28038780 0x7f9c2803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:20.946 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.302+0000 7f9c3cff9700 1 -- 172.21.15.115:0/717867458 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9c3004c4c0 con 0x7f9c40103ef0 2023-07-14T18:04:20.947 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.303+0000 7f9c3effd700 1 --2- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f9c28038780 0x7f9c2803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:20.947 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.303+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c40061c70 con 0x7f9c40103ef0 2023-07-14T18:04:20.947 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.303+0000 7f9c3effd700 1 --2- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f9c28038780 0x7f9c2803ac30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9c34006fd0 tx=0x7f9c34006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:20.947 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.307+0000 7f9c3cff9700 1 -- 172.21.15.115:0/717867458 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9c300187c0 con 0x7f9c40103ef0 2023-07-14T18:04:20.947 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.507+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f9c40105d80 con 0x7f9c40103ef0 2023-07-14T18:04:20.947 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.512+0000 7f9c3cff9700 1 -- 172.21.15.115:0/717867458 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f9c30022e60 con 0x7f9c40103ef0 2023-07-14T18:04:20.947 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f9c28038780 msgr2=0x7f9c2803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:20.948 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 --2- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f9c28038780 0x7f9c2803ac30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9c34006fd0 tx=0x7f9c34006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:20.948 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 msgr2=0x7f9c4018ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:20.948 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 --2- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c4018ef90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9c30006af0 tx=0x7f9c30011e80 comp rx=0 tx=0).stop 2023-07-14T18:04:20.948 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 shutdown_connections 2023-07-14T18:04:20.948 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 --2- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f9c28038780 0x7f9c2803ac30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:20.948 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 --2- 172.21.15.115:0/717867458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9c40103ef0 0x7f9c4018ef90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:20.948 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 >> 172.21.15.115:0/717867458 conn(0x7f9c400ff4f0 msgr2=0x7f9c40100230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:20.949 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 shutdown_connections 2023-07-14T18:04:20.949 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:20.515+0000 7f9c44fa1700 1 -- 172.21.15.115:0/717867458 wait complete. 2023-07-14T18:04:20.949 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling the dashboard module... 2023-07-14T18:04:21.595 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:21 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:21.595 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:21 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/717867458' entity='client.admin' 2023-07-14T18:04:21.596 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:21 smithi115 ceph-mon[127451]: from='mgr.14118 172.21.15.115:0/1004457596' entity='mgr.x' 2023-07-14T18:04:22.489 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:22 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3516125709' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-14T18:04:22.489 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: ignoring --setuser ceph since I am not root 2023-07-14T18:04:22.489 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: ignoring --setgroup ceph since I am not root 2023-07-14T18:04:22.490 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:22.379+0000 7f46756bb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:04:22.490 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:22.460+0000 7f46756bb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:04:22.783 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.395+0000 7f87e664c700 1 Processor -- start 2023-07-14T18:04:22.783 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.395+0000 7f87e664c700 1 -- start start 2023-07-14T18:04:22.783 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.395+0000 7f87e664c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0104480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:22.783 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.395+0000 7f87e664c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e01049c0 con 0x7f87e0104070 2023-07-14T18:04:22.783 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.396+0000 7f87dffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0104480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:22.784 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.396+0000 7f87dffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0104480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51442/0 (socket says 172.21.15.115:51442) 2023-07-14T18:04:22.784 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.396+0000 7f87dffff700 1 -- 172.21.15.115:0/3596848000 learned_addr learned my addr 172.21.15.115:0/3596848000 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:22.784 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.396+0000 7f87dffff700 1 -- 172.21.15.115:0/3596848000 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0104b00 con 0x7f87e0104070 2023-07-14T18:04:22.784 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.396+0000 7f87dffff700 1 --2- 172.21.15.115:0/3596848000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0104480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f87d40098a0 tx=0x7f87d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6372a62aea2d7609 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:22.784 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.396+0000 7f87deffd700 1 -- 172.21.15.115:0/3596848000 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87d4004160 con 0x7f87e0104070 2023-07-14T18:04:22.787 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.396+0000 7f87deffd700 1 -- 172.21.15.115:0/3596848000 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f87d40042c0 con 0x7f87e0104070 2023-07-14T18:04:22.787 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.396+0000 7f87deffd700 1 -- 172.21.15.115:0/3596848000 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87d40118e0 con 0x7f87e0104070 2023-07-14T18:04:22.788 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.397+0000 7f87e664c700 1 -- 172.21.15.115:0/3596848000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 msgr2=0x7f87e0104480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:22.788 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.397+0000 7f87e664c700 1 --2- 172.21.15.115:0/3596848000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0104480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f87d40098a0 tx=0x7f87d4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:22.788 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.397+0000 7f87e664c700 1 -- 172.21.15.115:0/3596848000 shutdown_connections 2023-07-14T18:04:22.788 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.397+0000 7f87e664c700 1 --2- 172.21.15.115:0/3596848000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0104480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:22.788 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.397+0000 7f87e664c700 1 -- 172.21.15.115:0/3596848000 >> 172.21.15.115:0/3596848000 conn(0x7f87e00ff6e0 msgr2=0x7f87e0101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:22.788 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.397+0000 7f87e664c700 1 -- 172.21.15.115:0/3596848000 shutdown_connections 2023-07-14T18:04:22.788 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.397+0000 7f87e664c700 1 -- 172.21.15.115:0/3596848000 wait complete. 2023-07-14T18:04:22.789 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.398+0000 7f87e664c700 1 Processor -- start 2023-07-14T18:04:22.789 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.398+0000 7f87e664c700 1 -- start start 2023-07-14T18:04:22.789 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.398+0000 7f87e664c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:22.789 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.398+0000 7f87e664c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e0198190 con 0x7f87e0104070 2023-07-14T18:04:22.789 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.398+0000 7f87dffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:22.789 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.398+0000 7f87dffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51444/0 (socket says 172.21.15.115:51444) 2023-07-14T18:04:22.789 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.398+0000 7f87dffff700 1 -- 172.21.15.115:0/3516125709 learned_addr learned my addr 172.21.15.115:0/3516125709 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:22.789 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.398+0000 7f87dffff700 1 -- 172.21.15.115:0/3516125709 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d4009550 con 0x7f87e0104070 2023-07-14T18:04:22.790 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.399+0000 7f87dffff700 1 --2- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0197c50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f87d40118f0 tx=0x7f87d40119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:22.790 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.399+0000 7f87dd7fa700 1 -- 172.21.15.115:0/3516125709 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87d40036a0 con 0x7f87e0104070 2023-07-14T18:04:22.790 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.399+0000 7f87dd7fa700 1 -- 172.21.15.115:0/3516125709 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f87d401b440 con 0x7f87e0104070 2023-07-14T18:04:22.790 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.399+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e0198390 con 0x7f87e0104070 2023-07-14T18:04:22.790 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.399+0000 7f87dd7fa700 1 -- 172.21.15.115:0/3516125709 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87d40193f0 con 0x7f87e0104070 2023-07-14T18:04:22.790 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.399+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e01987b0 con 0x7f87e0104070 2023-07-14T18:04:22.790 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.400+0000 7f87dd7fa700 1 -- 172.21.15.115:0/3516125709 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f87d4019660 con 0x7f87e0104070 2023-07-14T18:04:22.791 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.400+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e0061c70 con 0x7f87e0104070 2023-07-14T18:04:22.791 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.400+0000 7f87dd7fa700 1 --2- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f87c8038820 0x7f87c803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:22.791 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.400+0000 7f87dd7fa700 1 -- 172.21.15.115:0/3516125709 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f87d404c5a0 con 0x7f87e0104070 2023-07-14T18:04:22.791 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.400+0000 7f87df7fe700 1 --2- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f87c8038820 0x7f87c803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:22.791 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.400+0000 7f87df7fe700 1 --2- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f87c8038820 0x7f87c803acd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f87d0006fd0 tx=0x7f87d0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:22.791 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.404+0000 7f87dd7fa700 1 -- 172.21.15.115:0/3516125709 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f87d40239a0 con 0x7f87e0104070 2023-07-14T18:04:22.791 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:21.653+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f87e0105f10 con 0x7f87e0104070 2023-07-14T18:04:22.791 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.205+0000 7f87dd7fa700 1 -- 172.21.15.115:0/3516125709 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f87d401a430 con 0x7f87e0104070 2023-07-14T18:04:22.792 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.206+0000 7f87dd7fa700 1 -- 172.21.15.115:0/3516125709 <== mon.0 v2:172.21.15.115:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f87d4018740 con 0x7f87e0104070 2023-07-14T18:04:22.792 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.208+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f87c8038820 msgr2=0x7f87c803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:22.792 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.208+0000 7f87e664c700 1 --2- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f87c8038820 0x7f87c803acd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f87d0006fd0 tx=0x7f87d0006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:22.792 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.208+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 msgr2=0x7f87e0197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:22.792 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.208+0000 7f87e664c700 1 --2- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0197c50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f87d40118f0 tx=0x7f87d40119d0 comp rx=0 tx=0).stop 2023-07-14T18:04:22.792 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.208+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 shutdown_connections 2023-07-14T18:04:22.792 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.208+0000 7f87e664c700 1 --2- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f87c8038820 0x7f87c803acd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:22.793 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.208+0000 7f87e664c700 1 --2- 172.21.15.115:0/3516125709 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0104070 0x7f87e0197c50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:22.793 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.208+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 >> 172.21.15.115:0/3516125709 conn(0x7f87e00ff6e0 msgr2=0x7f87e0100330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:22.793 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.208+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 shutdown_connections 2023-07-14T18:04:22.793 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:22.209+0000 7f87e664c700 1 -- 172.21.15.115:0/3516125709 wait complete. 2023-07-14T18:04:22.845 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:22.553+0000 7f46756bb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:04:22.846 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:22.647+0000 7f46756bb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:04:23.183 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:22.834+0000 7f46756bb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:04:23.183 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:23 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:23.154+0000 7f46756bb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:04:23.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:23 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3516125709' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-14T18:04:23.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:23 smithi115 ceph-mon[127451]: mgrmap e9: x(active, since 15s) 2023-07-14T18:04:23.506 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:23 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:23.247+0000 7f46756bb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:04:23.507 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:23 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:23.381+0000 7f46756bb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:04:23.845 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:23 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:23.477+0000 7f46756bb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:04:23.990 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:04:23.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-14T18:04:23.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T18:04:23.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T18:04:23.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T18:04:23.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:04:23.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.279+0000 7f1249091700 1 Processor -- start 2023-07-14T18:04:23.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.279+0000 7f1249091700 1 -- start start 2023-07-14T18:04:23.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.279+0000 7f1249091700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f12441080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:23.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.279+0000 7f1249091700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244079d10 con 0x7f1244105cd0 2023-07-14T18:04:23.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.279+0000 7f1242d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f12441080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:23.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.279+0000 7f1242d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f12441080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51458/0 (socket says 172.21.15.115:51458) 2023-07-14T18:04:23.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.279+0000 7f1242d9d700 1 -- 172.21.15.115:0/2660551770 learned_addr learned my addr 172.21.15.115:0/2660551770 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:23.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.280+0000 7f1242d9d700 1 -- 172.21.15.115:0/2660551770 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1244079e50 con 0x7f1244105cd0 2023-07-14T18:04:23.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.280+0000 7f1242d9d700 1 --2- 172.21.15.115:0/2660551770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f12441080b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f12340098a0 tx=0x7f1234009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6940192639f4b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:23.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.280+0000 7f1241d9b700 1 -- 172.21.15.115:0/2660551770 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1234004160 con 0x7f1244105cd0 2023-07-14T18:04:23.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.280+0000 7f1241d9b700 1 -- 172.21.15.115:0/2660551770 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f12340042c0 con 0x7f1244105cd0 2023-07-14T18:04:23.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.280+0000 7f1241d9b700 1 -- 172.21.15.115:0/2660551770 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12340118e0 con 0x7f1244105cd0 2023-07-14T18:04:23.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.281+0000 7f1249091700 1 -- 172.21.15.115:0/2660551770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 msgr2=0x7f12441080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:23.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.281+0000 7f1249091700 1 --2- 172.21.15.115:0/2660551770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f12441080b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f12340098a0 tx=0x7f1234009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:23.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.281+0000 7f1249091700 1 -- 172.21.15.115:0/2660551770 shutdown_connections 2023-07-14T18:04:23.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.281+0000 7f1249091700 1 --2- 172.21.15.115:0/2660551770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f12441080b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:23.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.281+0000 7f1249091700 1 -- 172.21.15.115:0/2660551770 >> 172.21.15.115:0/2660551770 conn(0x7f12440ff710 msgr2=0x7f1244101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:23.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.281+0000 7f1249091700 1 -- 172.21.15.115:0/2660551770 shutdown_connections 2023-07-14T18:04:23.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.281+0000 7f1249091700 1 -- 172.21.15.115:0/2660551770 wait complete. 2023-07-14T18:04:23.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.281+0000 7f1249091700 1 Processor -- start 2023-07-14T18:04:23.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.282+0000 7f1249091700 1 -- start start 2023-07-14T18:04:23.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.282+0000 7f1249091700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f1244105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:23.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.282+0000 7f1249091700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244105b70 con 0x7f1244105cd0 2023-07-14T18:04:23.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.282+0000 7f1242d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f1244105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:23.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.282+0000 7f1242d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f1244105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51460/0 (socket says 172.21.15.115:51460) 2023-07-14T18:04:23.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.282+0000 7f1242d9d700 1 -- 172.21.15.115:0/1785337843 learned_addr learned my addr 172.21.15.115:0/1785337843 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:23.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.282+0000 7f1242d9d700 1 -- 172.21.15.115:0/1785337843 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1234009550 con 0x7f1244105cd0 2023-07-14T18:04:23.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.282+0000 7f1242d9d700 1 --2- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f1244105630 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1234006af0 tx=0x7f1234011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:23.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.283+0000 7f122bfff700 1 -- 172.21.15.115:0/1785337843 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12340037d0 con 0x7f1244105cd0 2023-07-14T18:04:23.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.283+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1244103ce0 con 0x7f1244105cd0 2023-07-14T18:04:24.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.283+0000 7f122bfff700 1 -- 172.21.15.115:0/1785337843 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f123401b440 con 0x7f1244105cd0 2023-07-14T18:04:24.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.283+0000 7f122bfff700 1 -- 172.21.15.115:0/1785337843 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1234019560 con 0x7f1244105cd0 2023-07-14T18:04:24.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.283+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1244104160 con 0x7f1244105cd0 2023-07-14T18:04:24.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.284+0000 7f122bfff700 1 -- 172.21.15.115:0/1785337843 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f1234019820 con 0x7f1244105cd0 2023-07-14T18:04:24.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.284+0000 7f122bfff700 1 --2- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f122c0387d0 0x7f122c03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:24.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.284+0000 7f122bfff700 1 -- 172.21.15.115:0/1785337843 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f123404c570 con 0x7f1244105cd0 2023-07-14T18:04:24.000 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.284+0000 7f124259c700 1 -- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f122c0387d0 msgr2=0x7f122c03ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/2169355495 2023-07-14T18:04:24.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.284+0000 7f124259c700 1 --2- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f122c0387d0 0x7f122c03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:04:24.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.284+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12441091e0 con 0x7f1244105cd0 2023-07-14T18:04:24.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.289+0000 7f122bfff700 1 -- 172.21.15.115:0/1785337843 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1234018d90 con 0x7f1244105cd0 2023-07-14T18:04:24.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.485+0000 7f124259c700 1 -- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f122c0387d0 msgr2=0x7f122c03ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/2169355495 2023-07-14T18:04:24.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.485+0000 7f124259c700 1 --2- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f122c0387d0 0x7f122c03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:04:24.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.537+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f124419edb0 con 0x7f1244105cd0 2023-07-14T18:04:24.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.538+0000 7f122bfff700 1 -- 172.21.15.115:0/1785337843 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f1234018350 con 0x7f1244105cd0 2023-07-14T18:04:24.001 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f122c0387d0 msgr2=0x7f122c03ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:04:24.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 --2- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f122c0387d0 0x7f122c03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:24.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 msgr2=0x7f1244105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:24.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 --2- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f1244105630 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1234006af0 tx=0x7f1234011e80 comp rx=0 tx=0).stop 2023-07-14T18:04:24.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 shutdown_connections 2023-07-14T18:04:24.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 --2- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f122c0387d0 0x7f122c03ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:24.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 --2- 172.21.15.115:0/1785337843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1244105cd0 0x7f1244105630 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:24.002 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 >> 172.21.15.115:0/1785337843 conn(0x7f12440ff710 msgr2=0x7f1244100430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:24.003 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 shutdown_connections 2023-07-14T18:04:24.003 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:23.540+0000 7f1249091700 1 -- 172.21.15.115:0/1785337843 wait complete. 2023-07-14T18:04:24.003 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for the mgr to restart... 2023-07-14T18:04:24.003 INFO:teuthology.orchestra.run.smithi115.stdout:Waiting for mgr epoch 9... 2023-07-14T18:04:24.213 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:23 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:23.912+0000 7f46756bb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:04:24.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:24 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1785337843' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T18:04:24.845 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:24 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:24.501+0000 7f46756bb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:04:24.846 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:24 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:24.669+0000 7f46756bb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:04:25.718 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:25.429+0000 7f46756bb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:04:25.719 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:25.519+0000 7f46756bb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:04:25.996 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:25.690+0000 7f46756bb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:04:26.345 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:25.967+0000 7f46756bb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:04:26.346 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:26 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:26.191+0000 7f46756bb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:04:26.628 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:26 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:26.338+0000 7f46756bb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:04:27.095 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:26 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:26.599+0000 7f46756bb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:04:27.595 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:27.293+0000 7f46756bb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:04:27.596 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:27.423+0000 7f46756bb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:04:27.971 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:27.613+0000 7f46756bb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:04:27.971 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:27.726+0000 7f46756bb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:04:27.971 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:27.942+0000 7f46756bb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:04:29.096 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:04:28.794+0000 7f46756bb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:04:29.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: Active manager daemon x restarted 2023-07-14T18:04:29.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: Activating manager daemon x 2023-07-14T18:04:29.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: osdmap e3: 0 total, 0 up, 0 in 2023-07-14T18:04:29.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: mgrmap e10: x(active, starting, since 0.00326118s) 2023-07-14T18:04:29.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:04:29.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T18:04:29.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T18:04:29.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T18:04:29.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T18:04:29.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: Manager daemon x is now available 2023-07-14T18:04:29.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:29.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:30.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T18:04:30.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T18:04:30.136 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:29 smithi115 ceph-mon[127451]: mgrmap e11: x(active, since 1.00543s) 2023-07-14T18:04:30.452 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:04:30.452 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-14T18:04:30.452 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T18:04:30.453 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:04:30.453 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.402+0000 7f930e5eb700 1 Processor -- start 2023-07-14T18:04:30.453 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.402+0000 7f930e5eb700 1 -- start start 2023-07-14T18:04:30.453 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.402+0000 7f930e5eb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f93081080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:30.453 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.402+0000 7f930e5eb700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9308079d10 con 0x7f9308105cd0 2023-07-14T18:04:30.453 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.402+0000 7f9307fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f93081080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:30.453 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.402+0000 7f9307fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f93081080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51468/0 (socket says 172.21.15.115:51468) 2023-07-14T18:04:30.454 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.402+0000 7f9307fff700 1 -- 172.21.15.115:0/3054972075 learned_addr learned my addr 172.21.15.115:0/3054972075 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:30.454 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.402+0000 7f9307fff700 1 -- 172.21.15.115:0/3054972075 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9308079e50 con 0x7f9308105cd0 2023-07-14T18:04:30.454 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.402+0000 7f9307fff700 1 --2- 172.21.15.115:0/3054972075 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f93081080b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f92f80098a0 tx=0x7f92f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9b2bf6625d4a81 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:30.454 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f9306ffd700 1 -- 172.21.15.115:0/3054972075 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92f8004160 con 0x7f9308105cd0 2023-07-14T18:04:30.454 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f9306ffd700 1 -- 172.21.15.115:0/3054972075 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f92f80042c0 con 0x7f9308105cd0 2023-07-14T18:04:30.454 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f9306ffd700 1 -- 172.21.15.115:0/3054972075 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92f80118e0 con 0x7f9308105cd0 2023-07-14T18:04:30.454 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f930e5eb700 1 -- 172.21.15.115:0/3054972075 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 msgr2=0x7f93081080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:30.455 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f930e5eb700 1 --2- 172.21.15.115:0/3054972075 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f93081080b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f92f80098a0 tx=0x7f92f8009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:30.455 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f930e5eb700 1 -- 172.21.15.115:0/3054972075 shutdown_connections 2023-07-14T18:04:30.455 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f930e5eb700 1 --2- 172.21.15.115:0/3054972075 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f93081080b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:30.455 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f930e5eb700 1 -- 172.21.15.115:0/3054972075 >> 172.21.15.115:0/3054972075 conn(0x7f93080ff710 msgr2=0x7f9308101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:30.455 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f930e5eb700 1 -- 172.21.15.115:0/3054972075 shutdown_connections 2023-07-14T18:04:30.455 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.403+0000 7f930e5eb700 1 -- 172.21.15.115:0/3054972075 wait complete. 2023-07-14T18:04:30.455 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.404+0000 7f930e5eb700 1 Processor -- start 2023-07-14T18:04:30.455 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.404+0000 7f930e5eb700 1 -- start start 2023-07-14T18:04:30.456 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.404+0000 7f930e5eb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f9308198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:30.456 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.404+0000 7f930e5eb700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93081992a0 con 0x7f9308105cd0 2023-07-14T18:04:30.456 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.404+0000 7f9307fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f9308198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:30.456 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.404+0000 7f9307fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f9308198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51470/0 (socket says 172.21.15.115:51470) 2023-07-14T18:04:30.456 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.404+0000 7f9307fff700 1 -- 172.21.15.115:0/3207422058 learned_addr learned my addr 172.21.15.115:0/3207422058 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:30.456 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.405+0000 7f9307fff700 1 -- 172.21.15.115:0/3207422058 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f8009550 con 0x7f9308105cd0 2023-07-14T18:04:30.456 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.405+0000 7f9307fff700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f9308198d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f92f8006af0 tx=0x7f92f8011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:30.457 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.405+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92f80037d0 con 0x7f9308105cd0 2023-07-14T18:04:30.457 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.405+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f92f801b440 con 0x7f9308105cd0 2023-07-14T18:04:30.457 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.405+0000 7f930e5eb700 1 -- 172.21.15.115:0/3207422058 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9308199500 con 0x7f9308105cd0 2023-07-14T18:04:30.457 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.405+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92f8019560 con 0x7f9308105cd0 2023-07-14T18:04:30.457 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.405+0000 7f930e5eb700 1 -- 172.21.15.115:0/3207422058 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9308199980 con 0x7f9308105cd0 2023-07-14T18:04:30.457 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.406+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f92f8019820 con 0x7f9308105cd0 2023-07-14T18:04:30.457 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.406+0000 7f93057fa700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 0x7f92f003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:30.457 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.406+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 --> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92f003b390 con 0x7f92f00387d0 2023-07-14T18:04:30.458 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.406+0000 7f93077fe700 1 -- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 msgr2=0x7f92f003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/2169355495 2023-07-14T18:04:30.458 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.406+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f92f804c790 con 0x7f9308105cd0 2023-07-14T18:04:30.458 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.406+0000 7f93077fe700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 0x7f92f003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:04:30.458 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.607+0000 7f93077fe700 1 -- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 msgr2=0x7f92f003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/2169355495 2023-07-14T18:04:30.458 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:24.607+0000 7f93077fe700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 0x7f92f003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:04:30.458 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:25.008+0000 7f93077fe700 1 -- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 msgr2=0x7f92f003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/2169355495 2023-07-14T18:04:30.458 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:25.008+0000 7f93077fe700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 0x7f92f003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T18:04:30.458 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:25.809+0000 7f93077fe700 1 -- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 msgr2=0x7f92f003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/2169355495 2023-07-14T18:04:30.459 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:25.809+0000 7f93077fe700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 0x7f92f003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T18:04:30.459 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:27.410+0000 7f93077fe700 1 -- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 msgr2=0x7f92f003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.115:6800/2169355495 2023-07-14T18:04:30.459 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:27.410+0000 7f93077fe700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 0x7f92f003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T18:04:30.459 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:28.809+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f92f8019b70 con 0x7f9308105cd0 2023-07-14T18:04:30.459 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:28.809+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 msgr2=0x7f92f003ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:04:30.459 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:28.809+0000 7f93057fa700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 0x7f92f003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:30.459 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.810+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f92f801a3d0 con 0x7f9308105cd0 2023-07-14T18:04:30.460 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.811+0000 7f93057fa700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92f003bfe0 0x7f92f003e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:30.460 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.811+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92f003b390 con 0x7f92f003bfe0 2023-07-14T18:04:30.460 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.811+0000 7f93077fe700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92f003bfe0 0x7f92f003e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:30.460 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.812+0000 7f93077fe700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92f003bfe0 0x7f92f003e3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f92fc003a10 tx=0x7f92fc0092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:30.460 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.812+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f92f003b390 con 0x7f92f003bfe0 2023-07-14T18:04:30.460 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.818+0000 7f930e5eb700 1 -- 172.21.15.115:0/3207422058 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f93081091e0 con 0x7f92f003bfe0 2023-07-14T18:04:30.460 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f93057fa700 1 -- 172.21.15.115:0/3207422058 <== mgr.14150 v2:172.21.15.115:6800/2625350855 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f93081091e0 con 0x7f92f003bfe0 2023-07-14T18:04:30.460 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 -- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92f003bfe0 msgr2=0x7f92f003e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:30.461 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92f003bfe0 0x7f92f003e3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f92fc003a10 tx=0x7f92fc0092b0 comp rx=0 tx=0).stop 2023-07-14T18:04:30.461 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 -- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 msgr2=0x7f9308198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:30.461 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f9308198d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f92f8006af0 tx=0x7f92f8011e80 comp rx=0 tx=0).stop 2023-07-14T18:04:30.461 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 -- 172.21.15.115:0/3207422058 shutdown_connections 2023-07-14T18:04:30.461 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92f003bfe0 0x7f92f003e3c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:30.461 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9308105cd0 0x7f9308198d60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:30.461 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 --2- 172.21.15.115:0/3207422058 >> [v2:172.21.15.115:6800/2169355495,v1:172.21.15.115:6801/2169355495] conn(0x7f92f00387d0 0x7f92f003ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:30.462 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 -- 172.21.15.115:0/3207422058 >> 172.21.15.115:0/3207422058 conn(0x7f93080ff710 msgr2=0x7f9308100430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:30.462 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 -- 172.21.15.115:0/3207422058 shutdown_connections 2023-07-14T18:04:30.462 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:29.819+0000 7f930e5eb700 1 -- 172.21.15.115:0/3207422058 wait complete. 2023-07-14T18:04:30.462 INFO:teuthology.orchestra.run.smithi115.stdout:mgr epoch 9 is available 2023-07-14T18:04:30.462 INFO:teuthology.orchestra.run.smithi115.stdout:Generating a dashboard self-signed certificate... 2023-07-14T18:04:31.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:30 smithi115 ceph-mon[127451]: [14/Jul/2023:18:04:29] ENGINE Bus STARTING 2023-07-14T18:04:31.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:30 smithi115 ceph-mon[127451]: [14/Jul/2023:18:04:29] ENGINE Serving on https://172.21.15.115:7150 2023-07-14T18:04:31.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:30 smithi115 ceph-mon[127451]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T18:04:31.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:30 smithi115 ceph-mon[127451]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T18:04:31.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:30 smithi115 ceph-mon[127451]: [14/Jul/2023:18:04:29] ENGINE Serving on http://172.21.15.115:8765 2023-07-14T18:04:31.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:30 smithi115 ceph-mon[127451]: [14/Jul/2023:18:04:29] ENGINE Bus STARTED 2023-07-14T18:04:31.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:31.645 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-14T18:04:31.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.948+0000 7ff4e2aba700 1 Processor -- start 2023-07-14T18:04:31.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.948+0000 7ff4e2aba700 1 -- start start 2023-07-14T18:04:31.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.948+0000 7ff4e2aba700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc1080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:31.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.948+0000 7ff4e2aba700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc079d10 con 0x7ff4dc105cd0 2023-07-14T18:04:31.646 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.948+0000 7ff4e0856700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc1080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:31.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.948+0000 7ff4e0856700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc1080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51524/0 (socket says 172.21.15.115:51524) 2023-07-14T18:04:31.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.948+0000 7ff4e0856700 1 -- 172.21.15.115:0/3128936188 learned_addr learned my addr 172.21.15.115:0/3128936188 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:31.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.948+0000 7ff4e0856700 1 -- 172.21.15.115:0/3128936188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc079e50 con 0x7ff4dc105cd0 2023-07-14T18:04:31.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4e0856700 1 --2- 172.21.15.115:0/3128936188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc1080b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00098a0 tx=0x7ff4d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6dbb7ed4abf690 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:31.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4db7fe700 1 -- 172.21.15.115:0/3128936188 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4d0004160 con 0x7ff4dc105cd0 2023-07-14T18:04:31.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4db7fe700 1 -- 172.21.15.115:0/3128936188 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff4d00042c0 con 0x7ff4dc105cd0 2023-07-14T18:04:31.647 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4db7fe700 1 -- 172.21.15.115:0/3128936188 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4d00118e0 con 0x7ff4dc105cd0 2023-07-14T18:04:31.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4e2aba700 1 -- 172.21.15.115:0/3128936188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 msgr2=0x7ff4dc1080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:31.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4e2aba700 1 --2- 172.21.15.115:0/3128936188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc1080b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00098a0 tx=0x7ff4d0009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:31.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4e2aba700 1 -- 172.21.15.115:0/3128936188 shutdown_connections 2023-07-14T18:04:31.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4e2aba700 1 --2- 172.21.15.115:0/3128936188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc1080b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:31.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4e2aba700 1 -- 172.21.15.115:0/3128936188 >> 172.21.15.115:0/3128936188 conn(0x7ff4dc0ff710 msgr2=0x7ff4dc101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:31.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4e2aba700 1 -- 172.21.15.115:0/3128936188 shutdown_connections 2023-07-14T18:04:31.648 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.949+0000 7ff4e2aba700 1 -- 172.21.15.115:0/3128936188 wait complete. 2023-07-14T18:04:31.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.950+0000 7ff4e2aba700 1 Processor -- start 2023-07-14T18:04:31.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.950+0000 7ff4e2aba700 1 -- start start 2023-07-14T18:04:31.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.950+0000 7ff4e2aba700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:31.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.950+0000 7ff4e2aba700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc1992a0 con 0x7ff4dc105cd0 2023-07-14T18:04:31.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.950+0000 7ff4e0856700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:31.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.950+0000 7ff4e0856700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51526/0 (socket says 172.21.15.115:51526) 2023-07-14T18:04:31.649 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.950+0000 7ff4e0856700 1 -- 172.21.15.115:0/2420985516 learned_addr learned my addr 172.21.15.115:0/2420985516 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:31.650 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.951+0000 7ff4e0856700 1 -- 172.21.15.115:0/2420985516 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d0009550 con 0x7ff4dc105cd0 2023-07-14T18:04:31.650 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.951+0000 7ff4e0856700 1 --2- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc198d60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0006af0 tx=0x7ff4d0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:31.650 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.951+0000 7ff4d9ffb700 1 -- 172.21.15.115:0/2420985516 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4d00037d0 con 0x7ff4dc105cd0 2023-07-14T18:04:31.650 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.951+0000 7ff4d9ffb700 1 -- 172.21.15.115:0/2420985516 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff4d001b440 con 0x7ff4dc105cd0 2023-07-14T18:04:31.650 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.951+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4dc199500 con 0x7ff4dc105cd0 2023-07-14T18:04:31.650 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.951+0000 7ff4d9ffb700 1 -- 172.21.15.115:0/2420985516 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4d0019560 con 0x7ff4dc105cd0 2023-07-14T18:04:31.651 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.951+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc199980 con 0x7ff4dc105cd0 2023-07-14T18:04:31.651 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.952+0000 7ff4d9ffb700 1 -- 172.21.15.115:0/2420985516 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7ff4d0003930 con 0x7ff4dc105cd0 2023-07-14T18:04:31.651 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.952+0000 7ff4d9ffb700 1 --2- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4cc0386b0 0x7ff4cc03ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:31.651 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.952+0000 7ff4d9ffb700 1 -- 172.21.15.115:0/2420985516 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff4d004c4b0 con 0x7ff4dc105cd0 2023-07-14T18:04:31.651 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.952+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc1091e0 con 0x7ff4dc105cd0 2023-07-14T18:04:31.652 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.952+0000 7ff4dbfff700 1 --2- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4cc0386b0 0x7ff4cc03ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:31.652 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.953+0000 7ff4dbfff700 1 --2- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4cc0386b0 0x7ff4cc03ab60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8006fd0 tx=0x7ff4c8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:31.652 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:30.957+0000 7ff4d9ffb700 1 -- 172.21.15.115:0/2420985516 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff4d0019820 con 0x7ff4dc105cd0 2023-07-14T18:04:31.652 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.161+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4dc100b40 con 0x7ff4cc0386b0 2023-07-14T18:04:31.652 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.291+0000 7ff4d9ffb700 1 -- 172.21.15.115:0/2420985516 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7ff4dc100b40 con 0x7ff4cc0386b0 2023-07-14T18:04:31.652 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.293+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4cc0386b0 msgr2=0x7ff4cc03ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:31.653 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.293+0000 7ff4e2aba700 1 --2- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4cc0386b0 0x7ff4cc03ab60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8006fd0 tx=0x7ff4c8006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:31.653 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.293+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 msgr2=0x7ff4dc198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:31.653 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.293+0000 7ff4e2aba700 1 --2- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc198d60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0006af0 tx=0x7ff4d0011e80 comp rx=0 tx=0).stop 2023-07-14T18:04:31.653 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.293+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 shutdown_connections 2023-07-14T18:04:31.653 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.293+0000 7ff4e2aba700 1 --2- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4cc0386b0 0x7ff4cc03ab60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:31.653 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.293+0000 7ff4e2aba700 1 --2- 172.21.15.115:0/2420985516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4dc105cd0 0x7ff4dc198d60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:31.653 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.293+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 >> 172.21.15.115:0/2420985516 conn(0x7ff4dc0ff710 msgr2=0x7ff4dc100430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:31.654 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.294+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 shutdown_connections 2023-07-14T18:04:31.654 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:31.294+0000 7ff4e2aba700 1 -- 172.21.15.115:0/2420985516 wait complete. 2023-07-14T18:04:31.654 INFO:teuthology.orchestra.run.smithi115.stdout:Creating initial admin user... 2023-07-14T18:04:32.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:31 smithi115 ceph-mon[127451]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:32.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:32.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:32.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:32.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:31 smithi115 ceph-mon[127451]: mgrmap e12: x(active, since 2s) 2023-07-14T18:04:32.991 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$l4uTO0CJHani7hRlWVO26e4lL3coXZXsGGRpWU453xWIZq7i.F8wW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689357872, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-14T18:04:32.991 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.049+0000 7fb5c280f700 1 Processor -- start 2023-07-14T18:04:32.991 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.049+0000 7fb5c280f700 1 -- start start 2023-07-14T18:04:32.992 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.049+0000 7fb5c280f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc104490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:32.992 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.049+0000 7fb5c280f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc1049d0 con 0x7fb5bc104080 2023-07-14T18:04:32.992 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.049+0000 7fb5bbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc104490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:32.992 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.049+0000 7fb5bbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc104490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51530/0 (socket says 172.21.15.115:51530) 2023-07-14T18:04:32.992 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.049+0000 7fb5bbfff700 1 -- 172.21.15.115:0/2807208578 learned_addr learned my addr 172.21.15.115:0/2807208578 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:32.992 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5bbfff700 1 -- 172.21.15.115:0/2807208578 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc104b10 con 0x7fb5bc104080 2023-07-14T18:04:32.993 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5bbfff700 1 --2- 172.21.15.115:0/2807208578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc104490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0098a0 tx=0x7fb5ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ce96ee2ef6c736 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:32.993 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5baffd700 1 -- 172.21.15.115:0/2807208578 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5ac004160 con 0x7fb5bc104080 2023-07-14T18:04:32.993 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5baffd700 1 -- 172.21.15.115:0/2807208578 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb5ac0042c0 con 0x7fb5bc104080 2023-07-14T18:04:32.993 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5baffd700 1 -- 172.21.15.115:0/2807208578 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5ac0118e0 con 0x7fb5bc104080 2023-07-14T18:04:32.993 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5c280f700 1 -- 172.21.15.115:0/2807208578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 msgr2=0x7fb5bc104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:32.993 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5c280f700 1 --2- 172.21.15.115:0/2807208578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc104490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0098a0 tx=0x7fb5ac009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:32.993 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5c280f700 1 -- 172.21.15.115:0/2807208578 shutdown_connections 2023-07-14T18:04:32.994 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5c280f700 1 --2- 172.21.15.115:0/2807208578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc104490 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:32.994 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.050+0000 7fb5c280f700 1 -- 172.21.15.115:0/2807208578 >> 172.21.15.115:0/2807208578 conn(0x7fb5bc0ff710 msgr2=0x7fb5bc101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:32.994 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.051+0000 7fb5c280f700 1 -- 172.21.15.115:0/2807208578 shutdown_connections 2023-07-14T18:04:32.994 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.051+0000 7fb5c280f700 1 -- 172.21.15.115:0/2807208578 wait complete. 2023-07-14T18:04:32.994 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.051+0000 7fb5c280f700 1 Processor -- start 2023-07-14T18:04:32.994 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.051+0000 7fb5c280f700 1 -- start start 2023-07-14T18:04:32.994 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.051+0000 7fb5c280f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:32.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.051+0000 7fb5c280f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc199310 con 0x7fb5bc104080 2023-07-14T18:04:32.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5bbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:32.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5bbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51532/0 (socket says 172.21.15.115:51532) 2023-07-14T18:04:32.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5bbfff700 1 -- 172.21.15.115:0/7534427 learned_addr learned my addr 172.21.15.115:0/7534427 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:32.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5bbfff700 1 -- 172.21.15.115:0/7534427 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac009550 con 0x7fb5bc104080 2023-07-14T18:04:32.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5bbfff700 1 --2- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc198dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0118f0 tx=0x7fb5ac0119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:32.995 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5b97fa700 1 -- 172.21.15.115:0/7534427 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5ac0036a0 con 0x7fb5bc104080 2023-07-14T18:04:32.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5b97fa700 1 -- 172.21.15.115:0/7534427 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb5ac01b440 con 0x7fb5bc104080 2023-07-14T18:04:32.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5bc199510 con 0x7fb5bc104080 2023-07-14T18:04:32.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5b97fa700 1 -- 172.21.15.115:0/7534427 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5ac0193f0 con 0x7fb5bc104080 2023-07-14T18:04:32.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.052+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5bc199930 con 0x7fb5bc104080 2023-07-14T18:04:32.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.053+0000 7fb5b97fa700 1 -- 172.21.15.115:0/7534427 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fb5ac003800 con 0x7fb5bc104080 2023-07-14T18:04:32.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.053+0000 7fb5b97fa700 1 --2- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb5a4038820 0x7fb5a403acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:32.996 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.053+0000 7fb5b97fa700 1 -- 172.21.15.115:0/7534427 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb5ac04c6f0 con 0x7fb5bc104080 2023-07-14T18:04:32.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.053+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5bc061c70 con 0x7fb5bc104080 2023-07-14T18:04:32.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.053+0000 7fb5bb7fe700 1 --2- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb5a4038820 0x7fb5a403acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:32.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.054+0000 7fb5bb7fe700 1 --2- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb5a4038820 0x7fb5a403acd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0006fd0 tx=0x7fb5b0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:32.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.058+0000 7fb5b97fa700 1 -- 172.21.15.115:0/7534427 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb5ac01f020 con 0x7fb5bc104080 2023-07-14T18:04:32.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.270+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb5bc061a90 con 0x7fb5a4038820 2023-07-14T18:04:32.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.494+0000 7fb5b97fa700 1 -- 172.21.15.115:0/7534427 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb5bc061a90 con 0x7fb5a4038820 2023-07-14T18:04:32.997 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb5a4038820 msgr2=0x7fb5a403acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:32.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 --2- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb5a4038820 0x7fb5a403acd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0006fd0 tx=0x7fb5b0006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:32.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 msgr2=0x7fb5bc198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:32.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 --2- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc198dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0118f0 tx=0x7fb5ac0119d0 comp rx=0 tx=0).stop 2023-07-14T18:04:32.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 shutdown_connections 2023-07-14T18:04:32.998 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 --2- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb5a4038820 0x7fb5a403acd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:32.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 --2- 172.21.15.115:0/7534427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb5bc104080 0x7fb5bc198dd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:32.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 >> 172.21.15.115:0/7534427 conn(0x7fb5bc0ff710 msgr2=0x7fb5bc0798b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:32.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 shutdown_connections 2023-07-14T18:04:32.999 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:32.497+0000 7fb5c280f700 1 -- 172.21.15.115:0/7534427 wait complete. 2023-07-14T18:04:32.999 INFO:teuthology.orchestra.run.smithi115.stdout:Fetching dashboard port number... 2023-07-14T18:04:33.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:33 smithi115 ceph-mon[127451]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:33.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:34.147 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stdout 8443 2023-07-14T18:04:34.147 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.524+0000 7f53821d7700 1 Processor -- start 2023-07-14T18:04:34.147 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.524+0000 7f53821d7700 1 -- start start 2023-07-14T18:04:34.147 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.524+0000 7f53821d7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c1080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:34.147 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.524+0000 7f53821d7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c079d10 con 0x7f537c105cc0 2023-07-14T18:04:34.147 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.524+0000 7f537b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c1080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:34.148 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.524+0000 7f537b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c1080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51536/0 (socket says 172.21.15.115:51536) 2023-07-14T18:04:34.148 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.524+0000 7f537b7fe700 1 -- 172.21.15.115:0/4046416144 learned_addr learned my addr 172.21.15.115:0/4046416144 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:34.148 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f537b7fe700 1 -- 172.21.15.115:0/4046416144 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c079e50 con 0x7f537c105cc0 2023-07-14T18:04:34.148 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f537b7fe700 1 --2- 172.21.15.115:0/4046416144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c1080a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f53700098a0 tx=0x7f5370009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3336ee9b69fb4348 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:34.148 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f537a7fc700 1 -- 172.21.15.115:0/4046416144 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5370004160 con 0x7f537c105cc0 2023-07-14T18:04:34.148 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f537a7fc700 1 -- 172.21.15.115:0/4046416144 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f53700042c0 con 0x7f537c105cc0 2023-07-14T18:04:34.148 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f537a7fc700 1 -- 172.21.15.115:0/4046416144 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53700118e0 con 0x7f537c105cc0 2023-07-14T18:04:34.149 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f53821d7700 1 -- 172.21.15.115:0/4046416144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 msgr2=0x7f537c1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:34.149 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f53821d7700 1 --2- 172.21.15.115:0/4046416144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c1080a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f53700098a0 tx=0x7f5370009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:34.149 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f53821d7700 1 -- 172.21.15.115:0/4046416144 shutdown_connections 2023-07-14T18:04:34.149 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f53821d7700 1 --2- 172.21.15.115:0/4046416144 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c1080a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:34.149 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f53821d7700 1 -- 172.21.15.115:0/4046416144 >> 172.21.15.115:0/4046416144 conn(0x7f537c0ff700 msgr2=0x7f537c101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:34.149 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.525+0000 7f53821d7700 1 -- 172.21.15.115:0/4046416144 shutdown_connections 2023-07-14T18:04:34.149 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.526+0000 7f53821d7700 1 -- 172.21.15.115:0/4046416144 wait complete. 2023-07-14T18:04:34.149 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.526+0000 7f53821d7700 1 Processor -- start 2023-07-14T18:04:34.150 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.526+0000 7f53821d7700 1 -- start start 2023-07-14T18:04:34.150 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.526+0000 7f53821d7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:34.150 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.526+0000 7f53821d7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c199260 con 0x7f537c105cc0 2023-07-14T18:04:34.150 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.526+0000 7f537b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:34.150 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.526+0000 7f537b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51538/0 (socket says 172.21.15.115:51538) 2023-07-14T18:04:34.150 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.526+0000 7f537b7fe700 1 -- 172.21.15.115:0/4151229167 learned_addr learned my addr 172.21.15.115:0/4151229167 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:34.150 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.527+0000 7f537b7fe700 1 -- 172.21.15.115:0/4151229167 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5370009550 con 0x7f537c105cc0 2023-07-14T18:04:34.150 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.527+0000 7f537b7fe700 1 --2- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c198d20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5370000c00 tx=0x7f53700119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:34.151 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.527+0000 7f5378ff9700 1 -- 172.21.15.115:0/4151229167 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53700036a0 con 0x7f537c105cc0 2023-07-14T18:04:34.151 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.527+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537c199460 con 0x7f537c105cc0 2023-07-14T18:04:34.151 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.527+0000 7f5378ff9700 1 -- 172.21.15.115:0/4151229167 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f537001b440 con 0x7f537c105cc0 2023-07-14T18:04:34.151 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.527+0000 7f5378ff9700 1 -- 172.21.15.115:0/4151229167 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f537001a3d0 con 0x7f537c105cc0 2023-07-14T18:04:34.151 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.527+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537c1998e0 con 0x7f537c105cc0 2023-07-14T18:04:34.151 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.528+0000 7f5378ff9700 1 -- 172.21.15.115:0/4151229167 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f537001a690 con 0x7f537c105cc0 2023-07-14T18:04:34.151 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.528+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537c1091d0 con 0x7f537c105cc0 2023-07-14T18:04:34.152 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.528+0000 7f5378ff9700 1 --2- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5364038820 0x7f536403acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:34.152 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.528+0000 7f5378ff9700 1 -- 172.21.15.115:0/4151229167 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f537004c900 con 0x7f537c105cc0 2023-07-14T18:04:34.152 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.528+0000 7f537affd700 1 --2- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5364038820 0x7f536403acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:34.152 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.528+0000 7f537affd700 1 --2- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5364038820 0x7f536403acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f536c00ad30 tx=0x7f536c0093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:34.152 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.533+0000 7f5378ff9700 1 -- 172.21.15.115:0/4151229167 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5370018b00 con 0x7f537c105cc0 2023-07-14T18:04:34.152 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.718+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f537c1093c0 con 0x7f537c105cc0 2023-07-14T18:04:34.152 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.719+0000 7f5378ff9700 1 -- 172.21.15.115:0/4151229167 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5370018b00 con 0x7f537c105cc0 2023-07-14T18:04:34.152 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.720+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5364038820 msgr2=0x7f536403acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:34.153 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.720+0000 7f53821d7700 1 --2- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5364038820 0x7f536403acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f536c00ad30 tx=0x7f536c0093f0 comp rx=0 tx=0).stop 2023-07-14T18:04:34.153 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.720+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 msgr2=0x7f537c198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:34.153 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.720+0000 7f53821d7700 1 --2- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c198d20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5370000c00 tx=0x7f53700119a0 comp rx=0 tx=0).stop 2023-07-14T18:04:34.153 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.721+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 shutdown_connections 2023-07-14T18:04:34.153 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.721+0000 7f53821d7700 1 --2- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5364038820 0x7f536403acd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:34.153 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.721+0000 7f53821d7700 1 --2- 172.21.15.115:0/4151229167 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f537c105cc0 0x7f537c198d20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:34.153 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.721+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 >> 172.21.15.115:0/4151229167 conn(0x7f537c0ff700 msgr2=0x7f537c1003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:34.153 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.721+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 shutdown_connections 2023-07-14T18:04:34.154 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:33.721+0000 7f53821d7700 1 -- 172.21.15.115:0/4151229167 wait complete. 2023-07-14T18:04:34.154 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:04:34.155 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout disabled 2023-07-14T18:04:34.161 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:04:34.161 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stdout inactive 2023-07-14T18:04:34.162 INFO:teuthology.orchestra.run.smithi115.stdout:firewalld.service is not enabled 2023-07-14T18:04:34.162 INFO:teuthology.orchestra.run.smithi115.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-14T18:04:34.163 INFO:teuthology.orchestra.run.smithi115.stdout:Ceph Dashboard is now available at: 2023-07-14T18:04:34.163 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:34.163 INFO:teuthology.orchestra.run.smithi115.stdout: URL: https://smithi115:8443/ 2023-07-14T18:04:34.163 INFO:teuthology.orchestra.run.smithi115.stdout: User: admin 2023-07-14T18:04:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: Password: emolty1uwd 2023-07-14T18:04:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:34.164 INFO:teuthology.orchestra.run.smithi115.stdout:Saving cluster configuration to /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config directory 2023-07-14T18:04:34.164 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling autotune for osd_memory_target 2023-07-14T18:04:34.836 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:34 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4151229167' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-14T18:04:35.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.586+0000 7ff9ce1c5700 1 Processor -- start 2023-07-14T18:04:35.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.586+0000 7ff9ce1c5700 1 -- start start 2023-07-14T18:04:35.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.586+0000 7ff9ce1c5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c81080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:35.207 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.586+0000 7ff9ce1c5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8079d10 con 0x7ff9c8105cc0 2023-07-14T18:04:35.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.587+0000 7ff9c77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c81080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:35.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.587+0000 7ff9c77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c81080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51546/0 (socket says 172.21.15.115:51546) 2023-07-14T18:04:35.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.587+0000 7ff9c77fe700 1 -- 172.21.15.115:0/4100270114 learned_addr learned my addr 172.21.15.115:0/4100270114 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:35.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.587+0000 7ff9c77fe700 1 -- 172.21.15.115:0/4100270114 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8079e50 con 0x7ff9c8105cc0 2023-07-14T18:04:35.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.587+0000 7ff9c77fe700 1 --2- 172.21.15.115:0/4100270114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c81080a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc0098a0 tx=0x7ff9bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b57b39fc84a095 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:35.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.587+0000 7ff9c67fc700 1 -- 172.21.15.115:0/4100270114 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9bc004160 con 0x7ff9c8105cc0 2023-07-14T18:04:35.208 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.587+0000 7ff9c67fc700 1 -- 172.21.15.115:0/4100270114 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff9bc0042c0 con 0x7ff9c8105cc0 2023-07-14T18:04:35.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.587+0000 7ff9c67fc700 1 -- 172.21.15.115:0/4100270114 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9bc0118e0 con 0x7ff9c8105cc0 2023-07-14T18:04:35.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.588+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/4100270114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 msgr2=0x7ff9c81080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:35.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.588+0000 7ff9ce1c5700 1 --2- 172.21.15.115:0/4100270114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c81080a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc0098a0 tx=0x7ff9bc009bb0 comp rx=0 tx=0).stop 2023-07-14T18:04:35.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.588+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/4100270114 shutdown_connections 2023-07-14T18:04:35.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.588+0000 7ff9ce1c5700 1 --2- 172.21.15.115:0/4100270114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c81080a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:35.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.588+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/4100270114 >> 172.21.15.115:0/4100270114 conn(0x7ff9c80ff700 msgr2=0x7ff9c8101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:35.209 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.588+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/4100270114 shutdown_connections 2023-07-14T18:04:35.210 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.588+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/4100270114 wait complete. 2023-07-14T18:04:35.210 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.588+0000 7ff9ce1c5700 1 Processor -- start 2023-07-14T18:04:35.210 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9ce1c5700 1 -- start start 2023-07-14T18:04:35.210 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9ce1c5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c8105620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:35.210 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9ce1c5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8105b60 con 0x7ff9c8105cc0 2023-07-14T18:04:35.210 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9c77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c8105620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:35.210 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9c77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c8105620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51548/0 (socket says 172.21.15.115:51548) 2023-07-14T18:04:35.210 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9c77fe700 1 -- 172.21.15.115:0/566886469 learned_addr learned my addr 172.21.15.115:0/566886469 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:35.211 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9c77fe700 1 -- 172.21.15.115:0/566886469 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc009550 con 0x7ff9c8105cc0 2023-07-14T18:04:35.211 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9c77fe700 1 --2- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c8105620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc000c00 tx=0x7ff9bc0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:35.211 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9c4ff9700 1 -- 172.21.15.115:0/566886469 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9bc0036a0 con 0x7ff9c8105cc0 2023-07-14T18:04:35.211 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9c4ff9700 1 -- 172.21.15.115:0/566886469 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff9bc01b440 con 0x7ff9c8105cc0 2023-07-14T18:04:35.211 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c8103c70 con 0x7ff9c8105cc0 2023-07-14T18:04:35.211 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.589+0000 7ff9c4ff9700 1 -- 172.21.15.115:0/566886469 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9bc01a3d0 con 0x7ff9c8105cc0 2023-07-14T18:04:35.211 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.590+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c81040f0 con 0x7ff9c8105cc0 2023-07-14T18:04:35.211 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.590+0000 7ff9c4ff9700 1 -- 172.21.15.115:0/566886469 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7ff9bc01a690 con 0x7ff9c8105cc0 2023-07-14T18:04:35.212 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.590+0000 7ff9c4ff9700 1 --2- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff9b0038820 0x7ff9b003acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:35.212 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.591+0000 7ff9c4ff9700 1 -- 172.21.15.115:0/566886469 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff9bc04c900 con 0x7ff9c8105cc0 2023-07-14T18:04:35.212 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.591+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c81091d0 con 0x7ff9c8105cc0 2023-07-14T18:04:35.212 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.591+0000 7ff9c6ffd700 1 --2- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff9b0038820 0x7ff9b003acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:35.212 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.591+0000 7ff9c6ffd700 1 --2- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff9b0038820 0x7ff9b003acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8006fd0 tx=0x7ff9b8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:35.212 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.595+0000 7ff9c4ff9700 1 -- 172.21.15.115:0/566886469 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff9bc018b00 con 0x7ff9c8105cc0 2023-07-14T18:04:35.212 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.782+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7ff9c81093c0 con 0x7ff9c8105cc0 2023-07-14T18:04:35.213 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.782+0000 7ff9c4ff9700 1 -- 172.21.15.115:0/566886469 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7ff9bc018b00 con 0x7ff9c8105cc0 2023-07-14T18:04:35.213 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff9b0038820 msgr2=0x7ff9b003acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:35.213 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 --2- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff9b0038820 0x7ff9b003acd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8006fd0 tx=0x7ff9b8006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:35.213 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 msgr2=0x7ff9c8105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:35.213 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 --2- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c8105620 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc000c00 tx=0x7ff9bc0119a0 comp rx=0 tx=0).stop 2023-07-14T18:04:35.213 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 shutdown_connections 2023-07-14T18:04:35.213 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 --2- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff9b0038820 0x7ff9b003acd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:35.213 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 --2- 172.21.15.115:0/566886469 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff9c8105cc0 0x7ff9c8105620 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:35.214 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 >> 172.21.15.115:0/566886469 conn(0x7ff9c80ff700 msgr2=0x7ff9c81003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:35.214 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 shutdown_connections 2023-07-14T18:04:35.214 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:34.784+0000 7ff9ce1c5700 1 -- 172.21.15.115:0/566886469 wait complete. 2023-07-14T18:04:36.253 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.607+0000 7f4e75166700 1 Processor -- start 2023-07-14T18:04:36.254 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.607+0000 7f4e75166700 1 -- start start 2023-07-14T18:04:36.254 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.607+0000 7f4e75166700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70104480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:36.254 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.607+0000 7f4e75166700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e701049c0 con 0x7f4e70104070 2023-07-14T18:04:36.254 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.607+0000 7f4e6ed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70104480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:36.254 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.607+0000 7f4e6ed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70104480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51552/0 (socket says 172.21.15.115:51552) 2023-07-14T18:04:36.254 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.607+0000 7f4e6ed9d700 1 -- 172.21.15.115:0/628162511 learned_addr learned my addr 172.21.15.115:0/628162511 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:36.254 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.607+0000 7f4e6ed9d700 1 -- 172.21.15.115:0/628162511 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70104b00 con 0x7f4e70104070 2023-07-14T18:04:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e6ed9d700 1 --2- 172.21.15.115:0/628162511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70104480 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000bb40 tx=0x7f4e6000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac27978685fe5e28 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e6dd9b700 1 -- 172.21.15.115:0/628162511 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e6000dbe0 con 0x7f4e70104070 2023-07-14T18:04:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e6dd9b700 1 -- 172.21.15.115:0/628162511 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e600044c0 con 0x7f4e70104070 2023-07-14T18:04:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e6dd9b700 1 -- 172.21.15.115:0/628162511 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e60003d50 con 0x7f4e70104070 2023-07-14T18:04:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e75166700 1 -- 172.21.15.115:0/628162511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 msgr2=0x7f4e70104480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:36.255 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e75166700 1 --2- 172.21.15.115:0/628162511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70104480 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000bb40 tx=0x7f4e6000be50 comp rx=0 tx=0).stop 2023-07-14T18:04:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e75166700 1 -- 172.21.15.115:0/628162511 shutdown_connections 2023-07-14T18:04:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e75166700 1 --2- 172.21.15.115:0/628162511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70104480 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e75166700 1 -- 172.21.15.115:0/628162511 >> 172.21.15.115:0/628162511 conn(0x7f4e700ff700 msgr2=0x7f4e70101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e75166700 1 -- 172.21.15.115:0/628162511 shutdown_connections 2023-07-14T18:04:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.608+0000 7f4e75166700 1 -- 172.21.15.115:0/628162511 wait complete. 2023-07-14T18:04:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.609+0000 7f4e75166700 1 Processor -- start 2023-07-14T18:04:36.256 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.609+0000 7f4e75166700 1 -- start start 2023-07-14T18:04:36.257 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.609+0000 7f4e75166700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:36.257 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.609+0000 7f4e75166700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e70197070 con 0x7f4e70104070 2023-07-14T18:04:36.257 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e6ed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:36.257 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e6ed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51554/0 (socket says 172.21.15.115:51554) 2023-07-14T18:04:36.257 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e6ed9d700 1 -- 172.21.15.115:0/1796276071 learned_addr learned my addr 172.21.15.115:0/1796276071 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:36.257 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e6ed9d700 1 -- 172.21.15.115:0/1796276071 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6000b7f0 con 0x7f4e70104070 2023-07-14T18:04:36.257 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e6ed9d700 1 --2- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70196b30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e60004160 tx=0x7f4e60004240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:36.258 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e57fff700 1 -- 172.21.15.115:0/1796276071 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e6001e070 con 0x7f4e70104070 2023-07-14T18:04:36.258 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e57fff700 1 -- 172.21.15.115:0/1796276071 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e60010b60 con 0x7f4e70104070 2023-07-14T18:04:36.258 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e70197270 con 0x7f4e70104070 2023-07-14T18:04:36.258 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e57fff700 1 -- 172.21.15.115:0/1796276071 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e60022b20 con 0x7f4e70104070 2023-07-14T18:04:36.258 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.610+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e70197690 con 0x7f4e70104070 2023-07-14T18:04:36.258 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.611+0000 7f4e57fff700 1 -- 172.21.15.115:0/1796276071 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f4e6002c440 con 0x7f4e70104070 2023-07-14T18:04:36.258 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.611+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e701909a0 con 0x7f4e70104070 2023-07-14T18:04:36.259 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.611+0000 7f4e57fff700 1 --2- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4e580388c0 0x7f4e5803ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:36.259 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.611+0000 7f4e57fff700 1 -- 172.21.15.115:0/1796276071 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4e6004f1d0 con 0x7f4e70104070 2023-07-14T18:04:36.259 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.611+0000 7f4e6e59c700 1 --2- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4e580388c0 0x7f4e5803ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:36.259 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.612+0000 7f4e6e59c700 1 --2- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4e580388c0 0x7f4e5803ad70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4e64006fd0 tx=0x7f4e64006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:36.259 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.616+0000 7f4e57fff700 1 -- 172.21.15.115:0/1796276071 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4e6002a950 con 0x7f4e70104070 2023-07-14T18:04:36.259 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.905+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f4e70061250 con 0x7f4e70104070 2023-07-14T18:04:36.259 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.908+0000 7f4e57fff700 1 -- 172.21.15.115:0/1796276071 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4e60027020 con 0x7f4e70104070 2023-07-14T18:04:36.260 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4e580388c0 msgr2=0x7f4e5803ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:36.260 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 --2- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4e580388c0 0x7f4e5803ad70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4e64006fd0 tx=0x7f4e64006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:36.260 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 msgr2=0x7f4e70196b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:36.260 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 --2- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70196b30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e60004160 tx=0x7f4e60004240 comp rx=0 tx=0).stop 2023-07-14T18:04:36.260 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 shutdown_connections 2023-07-14T18:04:36.260 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 --2- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4e580388c0 0x7f4e5803ad70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:36.260 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 --2- 172.21.15.115:0/1796276071 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e70104070 0x7f4e70196b30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:36.261 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 >> 172.21.15.115:0/1796276071 conn(0x7f4e700ff700 msgr2=0x7f4e7018dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:36.261 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 shutdown_connections 2023-07-14T18:04:36.261 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr 2023-07-14T18:04:35.910+0000 7f4e75166700 1 -- 172.21.15.115:0/1796276071 wait complete. 2023-07-14T18:04:36.261 INFO:teuthology.orchestra.run.smithi115.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-14T18:04:36.261 INFO:teuthology.orchestra.run.smithi115.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-14T18:04:36.261 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:36.261 INFO:teuthology.orchestra.run.smithi115.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:04:36.262 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:36.262 INFO:teuthology.orchestra.run.smithi115.stdout:Or, if you are only running a single cluster on this host: 2023-07-14T18:04:36.262 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:36.262 INFO:teuthology.orchestra.run.smithi115.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-14T18:04:36.262 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:36.262 INFO:teuthology.orchestra.run.smithi115.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-14T18:04:36.262 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:36.263 INFO:teuthology.orchestra.run.smithi115.stdout: ceph telemetry on 2023-07-14T18:04:36.263 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:36.263 INFO:teuthology.orchestra.run.smithi115.stdout:For more information see: 2023-07-14T18:04:36.263 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:36.263 INFO:teuthology.orchestra.run.smithi115.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-14T18:04:36.263 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:36.263 INFO:teuthology.orchestra.run.smithi115.stdout:Bootstrap complete. 2023-07-14T18:04:36.294 INFO:tasks.cephadm:Fetching config... 2023-07-14T18:04:36.294 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:04:36.295 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-14T18:04:36.311 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-14T18:04:36.311 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:04:36.311 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-14T18:04:36.368 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-14T18:04:36.368 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:04:36.369 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-14T18:04:36.434 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-14T18:04:36.434 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:04:36.434 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-14T18:04:36.490 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-14T18:04:36.491 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDg98lIYrnbK9S3Fx7eGxRMbSstfdjll8LuQQa0sN56n/yY5YbAKJUUbsCDX59Jhp3ZZcoPhoS1DI2qExikA6bk5Km46qjGRAACeLPpI1KrO+FdLeFBPcCIVBjfXXVsD8u0vSJUmvze+9ULrMWDdmq31msr3qqqTEv9kkMCBYXjEEr3WbhQwV8S18T0IhYfEAugAcvegjm1+PCVhTo1aNRVLU+uxSFoMsCrLki7EvW3xRBlTL+/r+6fiGrXvCxcFjHp6cgcBuQPzFfU/6VmVIzmECq8eWv1EYYT9Uu9Da7az/LnmPnD4r2ypjDFobJhJ8yA9D3Qd9Cuh/1I2B/G+rIxkYIhg6Lc2oFlBoYT+rxd+aJbehVUJtSFIkz1fPh6G5r6A/Hyj6Acx3UMe9EXxHpOX2r3Mf8A8HPEzAoD+qjn7nB23r8ovT3taDqJBigxSlLP1im1AWInM20VRe9vTkOyqnXxgX58SfFIYloi+9fXjWmcP0OdtDB10Cj6teOjkC0= ceph-a39f385c-2270-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T18:04:36.570 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDg98lIYrnbK9S3Fx7eGxRMbSstfdjll8LuQQa0sN56n/yY5YbAKJUUbsCDX59Jhp3ZZcoPhoS1DI2qExikA6bk5Km46qjGRAACeLPpI1KrO+FdLeFBPcCIVBjfXXVsD8u0vSJUmvze+9ULrMWDdmq31msr3qqqTEv9kkMCBYXjEEr3WbhQwV8S18T0IhYfEAugAcvegjm1+PCVhTo1aNRVLU+uxSFoMsCrLki7EvW3xRBlTL+/r+6fiGrXvCxcFjHp6cgcBuQPzFfU/6VmVIzmECq8eWv1EYYT9Uu9Da7az/LnmPnD4r2ypjDFobJhJ8yA9D3Qd9Cuh/1I2B/G+rIxkYIhg6Lc2oFlBoYT+rxd+aJbehVUJtSFIkz1fPh6G5r6A/Hyj6Acx3UMe9EXxHpOX2r3Mf8A8HPEzAoD+qjn7nB23r8ovT3taDqJBigxSlLP1im1AWInM20VRe9vTkOyqnXxgX58SfFIYloi+9fXjWmcP0OdtDB10Cj6teOjkC0= ceph-a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:04:36.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDg98lIYrnbK9S3Fx7eGxRMbSstfdjll8LuQQa0sN56n/yY5YbAKJUUbsCDX59Jhp3ZZcoPhoS1DI2qExikA6bk5Km46qjGRAACeLPpI1KrO+FdLeFBPcCIVBjfXXVsD8u0vSJUmvze+9ULrMWDdmq31msr3qqqTEv9kkMCBYXjEEr3WbhQwV8S18T0IhYfEAugAcvegjm1+PCVhTo1aNRVLU+uxSFoMsCrLki7EvW3xRBlTL+/r+6fiGrXvCxcFjHp6cgcBuQPzFfU/6VmVIzmECq8eWv1EYYT9Uu9Da7az/LnmPnD4r2ypjDFobJhJ8yA9D3Qd9Cuh/1I2B/G+rIxkYIhg6Lc2oFlBoYT+rxd+aJbehVUJtSFIkz1fPh6G5r6A/Hyj6Acx3UMe9EXxHpOX2r3Mf8A8HPEzAoD+qjn7nB23r8ovT3taDqJBigxSlLP1im1AWInM20VRe9vTkOyqnXxgX58SfFIYloi+9fXjWmcP0OdtDB10Cj6teOjkC0= ceph-a39f385c-2270-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T18:04:36.629 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDg98lIYrnbK9S3Fx7eGxRMbSstfdjll8LuQQa0sN56n/yY5YbAKJUUbsCDX59Jhp3ZZcoPhoS1DI2qExikA6bk5Km46qjGRAACeLPpI1KrO+FdLeFBPcCIVBjfXXVsD8u0vSJUmvze+9ULrMWDdmq31msr3qqqTEv9kkMCBYXjEEr3WbhQwV8S18T0IhYfEAugAcvegjm1+PCVhTo1aNRVLU+uxSFoMsCrLki7EvW3xRBlTL+/r+6fiGrXvCxcFjHp6cgcBuQPzFfU/6VmVIzmECq8eWv1EYYT9Uu9Da7az/LnmPnD4r2ypjDFobJhJ8yA9D3Qd9Cuh/1I2B/G+rIxkYIhg6Lc2oFlBoYT+rxd+aJbehVUJtSFIkz1fPh6G5r6A/Hyj6Acx3UMe9EXxHpOX2r3Mf8A8HPEzAoD+qjn7nB23r8ovT3taDqJBigxSlLP1im1AWInM20VRe9vTkOyqnXxgX58SfFIYloi+9fXjWmcP0OdtDB10Cj6teOjkC0= ceph-a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:04:36.642 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDg98lIYrnbK9S3Fx7eGxRMbSstfdjll8LuQQa0sN56n/yY5YbAKJUUbsCDX59Jhp3ZZcoPhoS1DI2qExikA6bk5Km46qjGRAACeLPpI1KrO+FdLeFBPcCIVBjfXXVsD8u0vSJUmvze+9ULrMWDdmq31msr3qqqTEv9kkMCBYXjEEr3WbhQwV8S18T0IhYfEAugAcvegjm1+PCVhTo1aNRVLU+uxSFoMsCrLki7EvW3xRBlTL+/r+6fiGrXvCxcFjHp6cgcBuQPzFfU/6VmVIzmECq8eWv1EYYT9Uu9Da7az/LnmPnD4r2ypjDFobJhJ8yA9D3Qd9Cuh/1I2B/G+rIxkYIhg6Lc2oFlBoYT+rxd+aJbehVUJtSFIkz1fPh6G5r6A/Hyj6Acx3UMe9EXxHpOX2r3Mf8A8HPEzAoD+qjn7nB23r8ovT3taDqJBigxSlLP1im1AWInM20VRe9vTkOyqnXxgX58SfFIYloi+9fXjWmcP0OdtDB10Cj6teOjkC0= ceph-a39f385c-2270-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T18:04:36.686 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDg98lIYrnbK9S3Fx7eGxRMbSstfdjll8LuQQa0sN56n/yY5YbAKJUUbsCDX59Jhp3ZZcoPhoS1DI2qExikA6bk5Km46qjGRAACeLPpI1KrO+FdLeFBPcCIVBjfXXVsD8u0vSJUmvze+9ULrMWDdmq31msr3qqqTEv9kkMCBYXjEEr3WbhQwV8S18T0IhYfEAugAcvegjm1+PCVhTo1aNRVLU+uxSFoMsCrLki7EvW3xRBlTL+/r+6fiGrXvCxcFjHp6cgcBuQPzFfU/6VmVIzmECq8eWv1EYYT9Uu9Da7az/LnmPnD4r2ypjDFobJhJ8yA9D3Qd9Cuh/1I2B/G+rIxkYIhg6Lc2oFlBoYT+rxd+aJbehVUJtSFIkz1fPh6G5r6A/Hyj6Acx3UMe9EXxHpOX2r3Mf8A8HPEzAoD+qjn7nB23r8ovT3taDqJBigxSlLP1im1AWInM20VRe9vTkOyqnXxgX58SfFIYloi+9fXjWmcP0OdtDB10Cj6teOjkC0= ceph-a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:04:36.699 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-14T18:04:36.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:36 smithi115 ceph-mon[127451]: mgrmap e13: x(active, since 6s) 2023-07-14T18:04:36.846 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:36 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1796276071' entity='client.admin' 2023-07-14T18:04:37.156 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:04:38.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.670+0000 7fd775e2f700 1 -- 172.21.15.115:0/2768640796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 msgr2=0x7fd7700fea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:38.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.670+0000 7fd775e2f700 1 --2- 172.21.15.115:0/2768640796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 0x7fd7700fea30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd760009910 tx=0x7fd760009c20 comp rx=0 tx=0).stop 2023-07-14T18:04:38.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.670+0000 7fd775e2f700 1 -- 172.21.15.115:0/2768640796 shutdown_connections 2023-07-14T18:04:38.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.670+0000 7fd775e2f700 1 --2- 172.21.15.115:0/2768640796 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 0x7fd7700fea30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:38.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.670+0000 7fd775e2f700 1 -- 172.21.15.115:0/2768640796 >> 172.21.15.115:0/2768640796 conn(0x7fd7700f9bd0 msgr2=0x7fd7700fc000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:38.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.670+0000 7fd775e2f700 1 -- 172.21.15.115:0/2768640796 shutdown_connections 2023-07-14T18:04:38.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.670+0000 7fd775e2f700 1 -- 172.21.15.115:0/2768640796 wait complete. 2023-07-14T18:04:38.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.671+0000 7fd775e2f700 1 Processor -- start 2023-07-14T18:04:38.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.671+0000 7fd775e2f700 1 -- start start 2023-07-14T18:04:38.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.671+0000 7fd775e2f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 0x7fd7700683d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:38.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.671+0000 7fd775e2f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7701974d0 con 0x7fd7700fe620 2023-07-14T18:04:38.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.671+0000 7fd76f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 0x7fd7700683d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:38.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.671+0000 7fd76f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 0x7fd7700683d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51560/0 (socket says 172.21.15.115:51560) 2023-07-14T18:04:38.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.671+0000 7fd76f7fe700 1 -- 172.21.15.115:0/2941651940 learned_addr learned my addr 172.21.15.115:0/2941651940 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:38.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.671+0000 7fd76f7fe700 1 -- 172.21.15.115:0/2941651940 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600095f0 con 0x7fd7700fe620 2023-07-14T18:04:38.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.672+0000 7fd76f7fe700 1 --2- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 0x7fd7700683d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd760006010 tx=0x7fd760005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:38.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.672+0000 7fd76cff9700 1 -- 172.21.15.115:0/2941651940 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd76000ba90 con 0x7fd7700fe620 2023-07-14T18:04:38.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.672+0000 7fd76cff9700 1 -- 172.21.15.115:0/2941651940 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd76000bbf0 con 0x7fd7700fe620 2023-07-14T18:04:38.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.672+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd770197610 con 0x7fd7700fe620 2023-07-14T18:04:38.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.672+0000 7fd76cff9700 1 -- 172.21.15.115:0/2941651940 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd760017440 con 0x7fd7700fe620 2023-07-14T18:04:38.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.672+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd770197ab0 con 0x7fd7700fe620 2023-07-14T18:04:38.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.673+0000 7fd76cff9700 1 -- 172.21.15.115:0/2941651940 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fd7600175a0 con 0x7fd7700fe620 2023-07-14T18:04:38.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.673+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77018d0a0 con 0x7fd7700fe620 2023-07-14T18:04:38.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.673+0000 7fd76cff9700 1 --2- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd758038800 0x7fd75803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:38.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.673+0000 7fd76cff9700 1 -- 172.21.15.115:0/2941651940 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd76004cce0 con 0x7fd7700fe620 2023-07-14T18:04:38.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.673+0000 7fd76effd700 1 --2- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd758038800 0x7fd75803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:38.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.673+0000 7fd76effd700 1 --2- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd758038800 0x7fd75803acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd764006fd0 tx=0x7fd764006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:38.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.678+0000 7fd76cff9700 1 -- 172.21.15.115:0/2941651940 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd76002ab10 con 0x7fd7700fe620 2023-07-14T18:04:38.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:38 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:38.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:38 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:38.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:38 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:04:38.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:38 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:38.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:38 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:38.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:38 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:38.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.881+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fd77002c930 con 0x7fd7700fe620 2023-07-14T18:04:38.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.885+0000 7fd76cff9700 1 -- 172.21.15.115:0/2941651940 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd760027070 con 0x7fd7700fe620 2023-07-14T18:04:38.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd758038800 msgr2=0x7fd75803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:38.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 --2- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd758038800 0x7fd75803acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd764006fd0 tx=0x7fd764006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:38.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 msgr2=0x7fd7700683d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:38.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 --2- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 0x7fd7700683d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd760006010 tx=0x7fd760005810 comp rx=0 tx=0).stop 2023-07-14T18:04:38.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 shutdown_connections 2023-07-14T18:04:38.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 --2- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd758038800 0x7fd75803acb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:38.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 --2- 172.21.15.115:0/2941651940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd7700fe620 0x7fd7700683d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:38.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 >> 172.21.15.115:0/2941651940 conn(0x7fd7700f9bd0 msgr2=0x7fd7700fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:38.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 shutdown_connections 2023-07-14T18:04:38.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:38.887+0000 7fd775e2f700 1 -- 172.21.15.115:0/2941651940 wait complete. 2023-07-14T18:04:39.398 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-14T18:04:39.398 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-14T18:04:39.949 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:04:40.186 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:39 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2941651940' entity='client.admin' 2023-07-14T18:04:40.187 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:39 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:40.187 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:39 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:41.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.227+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/1177472022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 msgr2=0x7f7bb01000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:41.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.227+0000 7f7bb4d1c700 1 --2- 172.21.15.115:0/1177472022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 0x7f7bb01000b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009910 tx=0x7f7ba4009c20 comp rx=0 tx=0).stop 2023-07-14T18:04:41.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.227+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/1177472022 shutdown_connections 2023-07-14T18:04:41.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.227+0000 7f7bb4d1c700 1 --2- 172.21.15.115:0/1177472022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 0x7f7bb01000b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:41.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.227+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/1177472022 >> 172.21.15.115:0/1177472022 conn(0x7f7bb00f9350 msgr2=0x7f7bb00fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:41.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.227+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/1177472022 shutdown_connections 2023-07-14T18:04:41.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.227+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/1177472022 wait complete. 2023-07-14T18:04:41.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7bb4d1c700 1 Processor -- start 2023-07-14T18:04:41.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7bb4d1c700 1 -- start start 2023-07-14T18:04:41.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7bb4d1c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 0x7f7bb00fe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:41.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7bb4d1c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb01974e0 con 0x7f7bb00ffca0 2023-07-14T18:04:41.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7bae59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 0x7f7bb00fe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:41.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7bae59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 0x7f7bb00fe2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51566/0 (socket says 172.21.15.115:51566) 2023-07-14T18:04:41.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7bae59c700 1 -- 172.21.15.115:0/2201847895 learned_addr learned my addr 172.21.15.115:0/2201847895 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:41.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7bae59c700 1 -- 172.21.15.115:0/2201847895 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba40095f0 con 0x7f7bb00ffca0 2023-07-14T18:04:41.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7bae59c700 1 --2- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 0x7f7bb00fe2e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4005790 tx=0x7f7ba4004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:41.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.228+0000 7f7b9f7fe700 1 -- 172.21.15.115:0/2201847895 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ba400b960 con 0x7f7bb00ffca0 2023-07-14T18:04:41.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.229+0000 7f7b9f7fe700 1 -- 172.21.15.115:0/2201847895 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7ba400f460 con 0x7f7bb00ffca0 2023-07-14T18:04:41.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.229+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb0197620 con 0x7f7bb00ffca0 2023-07-14T18:04:41.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.229+0000 7f7b9f7fe700 1 -- 172.21.15.115:0/2201847895 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ba4017440 con 0x7f7bb00ffca0 2023-07-14T18:04:41.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.229+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb0197ac0 con 0x7f7bb00ffca0 2023-07-14T18:04:41.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.229+0000 7f7b9f7fe700 1 -- 172.21.15.115:0/2201847895 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f7ba4017680 con 0x7f7bb00ffca0 2023-07-14T18:04:41.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.229+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb018d150 con 0x7f7bb00ffca0 2023-07-14T18:04:41.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.230+0000 7f7b9f7fe700 1 --2- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b980387b0 0x7f7b9803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:41.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.230+0000 7f7b9f7fe700 1 -- 172.21.15.115:0/2201847895 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ba404cbb0 con 0x7f7bb00ffca0 2023-07-14T18:04:41.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.230+0000 7f7badd9b700 1 --2- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b980387b0 0x7f7b9803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:41.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.230+0000 7f7badd9b700 1 --2- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b980387b0 0x7f7b9803ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0006fd0 tx=0x7f7ba0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:41.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.234+0000 7f7b9f7fe700 1 -- 172.21.15.115:0/2201847895 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7ba4015d90 con 0x7f7bb00ffca0 2023-07-14T18:04:41.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.426+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bb0060b10 con 0x7f7b980387b0 2023-07-14T18:04:41.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.429+0000 7f7b9f7fe700 1 -- 172.21.15.115:0/2201847895 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7bb0060b10 con 0x7f7b980387b0 2023-07-14T18:04:41.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b980387b0 msgr2=0x7f7b9803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:41.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 --2- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b980387b0 0x7f7b9803ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0006fd0 tx=0x7f7ba0006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:41.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 msgr2=0x7f7bb00fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:41.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 --2- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 0x7f7bb00fe2e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4005790 tx=0x7f7ba4004f90 comp rx=0 tx=0).stop 2023-07-14T18:04:41.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 shutdown_connections 2023-07-14T18:04:41.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 --2- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b980387b0 0x7f7b9803ac60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:41.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 --2- 172.21.15.115:0/2201847895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7bb00ffca0 0x7f7bb00fe2e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:41.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 >> 172.21.15.115:0/2201847895 conn(0x7f7bb00f9350 msgr2=0x7f7bb00fa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:41.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 shutdown_connections 2023-07-14T18:04:41.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:41.431+0000 7f7bb4d1c700 1 -- 172.21.15.115:0/2201847895 wait complete. 2023-07-14T18:04:42.000 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi150 2023-07-14T18:04:42.001 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T18:04:42.001 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.conf 2023-07-14T18:04:42.021 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T18:04:42.021 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:04:42.080 INFO:tasks.cephadm:Adding host smithi150 to orchestrator... 2023-07-14T18:04:42.080 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch host add smithi150 2023-07-14T18:04:42.641 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:04:42.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:42.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:42.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:42.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:04:42.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:04:42.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: Updating smithi115:/etc/ceph/ceph.conf 2023-07-14T18:04:42.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: Updating smithi115:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:04:42.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:04:42.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: Updating smithi115:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:04:42.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:42.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:42.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:42 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:43.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.781+0000 7f905e59b700 1 -- 172.21.15.115:0/862397654 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 msgr2=0x7f9058104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:43.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.781+0000 7f905e59b700 1 --2- 172.21.15.115:0/862397654 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 0x7f9058104a20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9048009910 tx=0x7f9048009c20 comp rx=0 tx=0).stop 2023-07-14T18:04:43.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.781+0000 7f905e59b700 1 -- 172.21.15.115:0/862397654 shutdown_connections 2023-07-14T18:04:43.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.781+0000 7f905e59b700 1 --2- 172.21.15.115:0/862397654 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 0x7f9058104a20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:43.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.781+0000 7f905e59b700 1 -- 172.21.15.115:0/862397654 >> 172.21.15.115:0/862397654 conn(0x7f90580743e0 msgr2=0x7f90580747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:43.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.781+0000 7f905e59b700 1 -- 172.21.15.115:0/862397654 shutdown_connections 2023-07-14T18:04:43.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.781+0000 7f905e59b700 1 -- 172.21.15.115:0/862397654 wait complete. 2023-07-14T18:04:43.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.782+0000 7f905e59b700 1 Processor -- start 2023-07-14T18:04:43.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.782+0000 7f905e59b700 1 -- start start 2023-07-14T18:04:43.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.782+0000 7f905e59b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 0x7f9058199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:43.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.782+0000 7f905e59b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9058199d80 con 0x7f9058104650 2023-07-14T18:04:43.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.782+0000 7f9057fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 0x7f9058199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:43.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.782+0000 7f9057fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 0x7f9058199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51572/0 (socket says 172.21.15.115:51572) 2023-07-14T18:04:43.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.782+0000 7f9057fff700 1 -- 172.21.15.115:0/3505630207 learned_addr learned my addr 172.21.15.115:0/3505630207 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:43.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.783+0000 7f9057fff700 1 -- 172.21.15.115:0/3505630207 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90480095f0 con 0x7f9058104650 2023-07-14T18:04:43.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.783+0000 7f9057fff700 1 --2- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 0x7f9058199840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9048005950 tx=0x7f9048004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:43.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.783+0000 7f90557fa700 1 -- 172.21.15.115:0/3505630207 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f904801d070 con 0x7f9058104650 2023-07-14T18:04:43.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.783+0000 7f90557fa700 1 -- 172.21.15.115:0/3505630207 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f904800bdf0 con 0x7f9058104650 2023-07-14T18:04:43.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.783+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9058199f80 con 0x7f9058104650 2023-07-14T18:04:43.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.783+0000 7f90557fa700 1 -- 172.21.15.115:0/3505630207 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9048022c50 con 0x7f9058104650 2023-07-14T18:04:43.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.783+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9058193830 con 0x7f9058104650 2023-07-14T18:04:43.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.784+0000 7f90557fa700 1 -- 172.21.15.115:0/3505630207 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f904800f460 con 0x7f9058104650 2023-07-14T18:04:43.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.784+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9058061c70 con 0x7f9058104650 2023-07-14T18:04:43.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.784+0000 7f90557fa700 1 --2- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f90440387b0 0x7f904403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:43.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.784+0000 7f90557fa700 1 -- 172.21.15.115:0/3505630207 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f904804cb10 con 0x7f9058104650 2023-07-14T18:04:43.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.784+0000 7f90577fe700 1 --2- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f90440387b0 0x7f904403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:43.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.784+0000 7f90577fe700 1 --2- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f90440387b0 0x7f904403ac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9040006fd0 tx=0x7f9040006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:43.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.789+0000 7f90557fa700 1 -- 172.21.15.115:0/3505630207 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f904802bba0 con 0x7f9058104650 2023-07-14T18:04:44.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:43.978+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}) v1 -- 0x7f9058194300 con 0x7f90440387b0 2023-07-14T18:04:44.596 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:44 smithi115 ceph-mon[127451]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:45.596 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:45 smithi115 ceph-mon[127451]: Deploying cephadm binary to smithi150 2023-07-14T18:04:46.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.238+0000 7f90557fa700 1 -- 172.21.15.115:0/3505630207 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9058194300 con 0x7f90440387b0 2023-07-14T18:04:46.268 INFO:teuthology.orchestra.run.smithi115.stdout:Added host 'smithi150' with addr '172.21.15.150' 2023-07-14T18:04:46.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.240+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f90440387b0 msgr2=0x7f904403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:46.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.240+0000 7f905e59b700 1 --2- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f90440387b0 0x7f904403ac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9040006fd0 tx=0x7f9040006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:46.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.240+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 msgr2=0x7f9058199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:46.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.240+0000 7f905e59b700 1 --2- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 0x7f9058199840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9048005950 tx=0x7f9048004ff0 comp rx=0 tx=0).stop 2023-07-14T18:04:46.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.240+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 shutdown_connections 2023-07-14T18:04:46.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.240+0000 7f905e59b700 1 --2- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f90440387b0 0x7f904403ac60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:46.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.240+0000 7f905e59b700 1 --2- 172.21.15.115:0/3505630207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9058104650 0x7f9058199840 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:46.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.240+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 >> 172.21.15.115:0/3505630207 conn(0x7f90580743e0 msgr2=0x7f9058105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:46.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.240+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 shutdown_connections 2023-07-14T18:04:46.271 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:46.241+0000 7f905e59b700 1 -- 172.21.15.115:0/3505630207 wait complete. 2023-07-14T18:04:46.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T18:04:47.245 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:04:47.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:47.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:47 smithi115 ceph-mon[127451]: Added host smithi150 2023-07-14T18:04:47.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:47.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:48.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.408+0000 7fd48e2b1700 1 -- 172.21.15.115:0/719846520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 msgr2=0x7fd488104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:48.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.408+0000 7fd48e2b1700 1 --2- 172.21.15.115:0/719846520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 0x7fd488104a40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd478009910 tx=0x7fd478009c20 comp rx=0 tx=0).stop 2023-07-14T18:04:48.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.408+0000 7fd48e2b1700 1 -- 172.21.15.115:0/719846520 shutdown_connections 2023-07-14T18:04:48.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.408+0000 7fd48e2b1700 1 --2- 172.21.15.115:0/719846520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 0x7fd488104a40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:48.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.408+0000 7fd48e2b1700 1 -- 172.21.15.115:0/719846520 >> 172.21.15.115:0/719846520 conn(0x7fd4880fa150 msgr2=0x7fd4880fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:48.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.408+0000 7fd48e2b1700 1 -- 172.21.15.115:0/719846520 shutdown_connections 2023-07-14T18:04:48.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.408+0000 7fd48e2b1700 1 -- 172.21.15.115:0/719846520 wait complete. 2023-07-14T18:04:48.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.409+0000 7fd48e2b1700 1 Processor -- start 2023-07-14T18:04:48.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.409+0000 7fd48e2b1700 1 -- start start 2023-07-14T18:04:48.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.409+0000 7fd48e2b1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 0x7fd4881996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:48.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.409+0000 7fd48e2b1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd488199c10 con 0x7fd488104670 2023-07-14T18:04:48.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.409+0000 7fd487fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 0x7fd4881996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:48.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.409+0000 7fd487fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 0x7fd4881996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51598/0 (socket says 172.21.15.115:51598) 2023-07-14T18:04:48.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.409+0000 7fd487fff700 1 -- 172.21.15.115:0/733351921 learned_addr learned my addr 172.21.15.115:0/733351921 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:48.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.410+0000 7fd487fff700 1 -- 172.21.15.115:0/733351921 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4780095f0 con 0x7fd488104670 2023-07-14T18:04:48.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.410+0000 7fd487fff700 1 --2- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 0x7fd4881996d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd478004f40 tx=0x7fd478005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:48.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.410+0000 7fd4857fa700 1 -- 172.21.15.115:0/733351921 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd47801d070 con 0x7fd488104670 2023-07-14T18:04:48.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.410+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd488199e70 con 0x7fd488104670 2023-07-14T18:04:48.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.410+0000 7fd4857fa700 1 -- 172.21.15.115:0/733351921 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd4780044b0 con 0x7fd488104670 2023-07-14T18:04:48.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.410+0000 7fd4857fa700 1 -- 172.21.15.115:0/733351921 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd478003810 con 0x7fd488104670 2023-07-14T18:04:48.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.410+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488193770 con 0x7fd488104670 2023-07-14T18:04:48.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.411+0000 7fd4857fa700 1 -- 172.21.15.115:0/733351921 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fd478003a90 con 0x7fd488104670 2023-07-14T18:04:48.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.411+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd488061c70 con 0x7fd488104670 2023-07-14T18:04:48.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.411+0000 7fd4857fa700 1 --2- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd4700387a0 0x7fd47003ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:48.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.411+0000 7fd4857fa700 1 -- 172.21.15.115:0/733351921 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd47804c8d0 con 0x7fd488104670 2023-07-14T18:04:48.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.411+0000 7fd4877fe700 1 --2- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd4700387a0 0x7fd47003ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:48.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.412+0000 7fd4877fe700 1 --2- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd4700387a0 0x7fd47003ac50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd47c006fd0 tx=0x7fd47c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:48.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.416+0000 7fd4857fa700 1 -- 172.21.15.115:0/733351921 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd47802a770 con 0x7fd488104670 2023-07-14T18:04:48.596 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:48 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:48.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.608+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd488194290 con 0x7fd4700387a0 2023-07-14T18:04:48.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.609+0000 7fd4857fa700 1 -- 172.21.15.115:0/733351921 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fd488194290 con 0x7fd4700387a0 2023-07-14T18:04:48.639 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:48.639 INFO:teuthology.orchestra.run.smithi115.stdout:[{"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}] 2023-07-14T18:04:48.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd4700387a0 msgr2=0x7fd47003ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:48.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 --2- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd4700387a0 0x7fd47003ac50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd47c006fd0 tx=0x7fd47c006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:48.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 msgr2=0x7fd4881996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:48.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 --2- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 0x7fd4881996d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd478004f40 tx=0x7fd478005020 comp rx=0 tx=0).stop 2023-07-14T18:04:48.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 shutdown_connections 2023-07-14T18:04:48.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 --2- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd4700387a0 0x7fd47003ac50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:48.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 --2- 172.21.15.115:0/733351921 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd488104670 0x7fd4881996d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:48.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 >> 172.21.15.115:0/733351921 conn(0x7fd4880fa150 msgr2=0x7fd4880fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:48.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 shutdown_connections 2023-07-14T18:04:48.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:48.611+0000 7fd48e2b1700 1 -- 172.21.15.115:0/733351921 wait complete. 2023-07-14T18:04:48.985 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi187 2023-07-14T18:04:48.986 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T18:04:48.986 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.conf 2023-07-14T18:04:49.006 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T18:04:49.006 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:04:49.066 INFO:tasks.cephadm:Adding host smithi187 to orchestrator... 2023-07-14T18:04:49.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch host add smithi187 2023-07-14T18:04:49.616 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:04:49.888 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:49.889 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:49 smithi115 ceph-mon[127451]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:04:49.889 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:49 smithi115 ceph-mon[127451]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:04:50.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.883+0000 7f5c95946700 1 -- 172.21.15.115:0/4039831398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 msgr2=0x7f5c900ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:50.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.883+0000 7f5c95946700 1 --2- 172.21.15.115:0/4039831398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 0x7f5c900ffa60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009910 tx=0x7f5c84009c20 comp rx=0 tx=0).stop 2023-07-14T18:04:50.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.883+0000 7f5c95946700 1 -- 172.21.15.115:0/4039831398 shutdown_connections 2023-07-14T18:04:50.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.883+0000 7f5c95946700 1 --2- 172.21.15.115:0/4039831398 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 0x7f5c900ffa60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:50.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.883+0000 7f5c95946700 1 -- 172.21.15.115:0/4039831398 >> 172.21.15.115:0/4039831398 conn(0x7f5c90075070 msgr2=0x7f5c90075470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:50.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.883+0000 7f5c95946700 1 -- 172.21.15.115:0/4039831398 shutdown_connections 2023-07-14T18:04:50.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.883+0000 7f5c95946700 1 -- 172.21.15.115:0/4039831398 wait complete. 2023-07-14T18:04:50.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.883+0000 7f5c95946700 1 Processor -- start 2023-07-14T18:04:50.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.883+0000 7f5c95946700 1 -- start start 2023-07-14T18:04:50.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c95946700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 0x7f5c90196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:50.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c95946700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c901972a0 con 0x7f5c900ff690 2023-07-14T18:04:50.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c8effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 0x7f5c90196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:50.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c8effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 0x7f5c90196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51604/0 (socket says 172.21.15.115:51604) 2023-07-14T18:04:50.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c8effd700 1 -- 172.21.15.115:0/20865721 learned_addr learned my addr 172.21.15.115:0/20865721 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:50.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c8effd700 1 -- 172.21.15.115:0/20865721 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c840095f0 con 0x7f5c900ff690 2023-07-14T18:04:50.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c8effd700 1 --2- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 0x7f5c90196d60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c84005950 tx=0x7f5c84004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:50.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c94944700 1 -- 172.21.15.115:0/20865721 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c8401d070 con 0x7f5c900ff690 2023-07-14T18:04:50.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c94944700 1 -- 172.21.15.115:0/20865721 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5c8400bdf0 con 0x7f5c900ff690 2023-07-14T18:04:50.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c901974a0 con 0x7f5c900ff690 2023-07-14T18:04:50.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c94944700 1 -- 172.21.15.115:0/20865721 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c84022c50 con 0x7f5c900ff690 2023-07-14T18:04:50.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.884+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c901978c0 con 0x7f5c900ff690 2023-07-14T18:04:50.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.885+0000 7f5c94944700 1 -- 172.21.15.115:0/20865721 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f5c84022470 con 0x7f5c900ff690 2023-07-14T18:04:50.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.885+0000 7f5c94944700 1 --2- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5c780387b0 0x7f5c7803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:50.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.885+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c90061c70 con 0x7f5c900ff690 2023-07-14T18:04:50.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.885+0000 7f5c94944700 1 -- 172.21.15.115:0/20865721 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5c8404cb20 con 0x7f5c900ff690 2023-07-14T18:04:50.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.885+0000 7f5c8e7fc700 1 --2- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5c780387b0 0x7f5c7803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:50.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.886+0000 7f5c8e7fc700 1 --2- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5c780387b0 0x7f5c7803ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5c80006fd0 tx=0x7f5c80006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:50.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:50.890+0000 7f5c94944700 1 -- 172.21.15.115:0/20865721 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5c84013250 con 0x7f5c900ff690 2023-07-14T18:04:51.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:51.085+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c901026e0 con 0x7f5c780387b0 2023-07-14T18:04:52.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:04:52.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:52.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:52.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:52.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:52.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:04:52.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:04:52.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:04:52.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: Updating smithi150:/etc/ceph/ceph.conf 2023-07-14T18:04:52.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:04:52.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:52.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:52.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:53.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:53 smithi115 ceph-mon[127451]: Updating smithi150:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:04:53.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:53 smithi115 ceph-mon[127451]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:04:53.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:53 smithi115 ceph-mon[127451]: Deploying cephadm binary to smithi187 2023-07-14T18:04:53.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:53 smithi115 ceph-mon[127451]: Updating smithi150:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:04:53.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.345+0000 7f5c94944700 1 -- 172.21.15.115:0/20865721 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5c901026e0 con 0x7f5c780387b0 2023-07-14T18:04:53.375 INFO:teuthology.orchestra.run.smithi115.stdout:Added host 'smithi187' with addr '172.21.15.187' 2023-07-14T18:04:53.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.347+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5c780387b0 msgr2=0x7f5c7803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:53.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.347+0000 7f5c95946700 1 --2- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5c780387b0 0x7f5c7803ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5c80006fd0 tx=0x7f5c80006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:53.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.348+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 msgr2=0x7f5c90196d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:53.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.348+0000 7f5c95946700 1 --2- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 0x7f5c90196d60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c84005950 tx=0x7f5c84004ff0 comp rx=0 tx=0).stop 2023-07-14T18:04:53.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.348+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 shutdown_connections 2023-07-14T18:04:53.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.348+0000 7f5c95946700 1 --2- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5c780387b0 0x7f5c7803ac60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:53.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.348+0000 7f5c95946700 1 --2- 172.21.15.115:0/20865721 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c900ff690 0x7f5c90196d60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:53.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.348+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 >> 172.21.15.115:0/20865721 conn(0x7f5c90075070 msgr2=0x7f5c900fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:53.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.348+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 shutdown_connections 2023-07-14T18:04:53.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:53.348+0000 7f5c95946700 1 -- 172.21.15.115:0/20865721 wait complete. 2023-07-14T18:04:53.775 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T18:04:54.325 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:04:54.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:54 smithi115 ceph-mon[127451]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:04:54.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:54.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:04:54.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:55.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:55 smithi115 ceph-mon[127451]: Added host smithi187 2023-07-14T18:04:55.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:55.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.621+0000 7f26a92f6700 1 -- 172.21.15.115:0/892801027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 msgr2=0x7f26a40ffc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:55.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.621+0000 7f26a92f6700 1 --2- 172.21.15.115:0/892801027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 0x7f26a40ffc50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2698009910 tx=0x7f2698009c20 comp rx=0 tx=0).stop 2023-07-14T18:04:55.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.622+0000 7f26a92f6700 1 -- 172.21.15.115:0/892801027 shutdown_connections 2023-07-14T18:04:55.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.622+0000 7f26a92f6700 1 --2- 172.21.15.115:0/892801027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 0x7f26a40ffc50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:55.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.622+0000 7f26a92f6700 1 -- 172.21.15.115:0/892801027 >> 172.21.15.115:0/892801027 conn(0x7f26a4074e40 msgr2=0x7f26a4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:55.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.622+0000 7f26a92f6700 1 -- 172.21.15.115:0/892801027 shutdown_connections 2023-07-14T18:04:55.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.622+0000 7f26a92f6700 1 -- 172.21.15.115:0/892801027 wait complete. 2023-07-14T18:04:55.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.622+0000 7f26a92f6700 1 Processor -- start 2023-07-14T18:04:55.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.622+0000 7f26a92f6700 1 -- start start 2023-07-14T18:04:55.655 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f26a92f6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 0x7f26a419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:55.655 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f26a92f6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a419cff0 con 0x7f26a40ff880 2023-07-14T18:04:55.655 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f26a2ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 0x7f26a419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:55.655 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f26a2ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 0x7f26a419cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51630/0 (socket says 172.21.15.115:51630) 2023-07-14T18:04:55.655 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f26a2ffd700 1 -- 172.21.15.115:0/180210503 learned_addr learned my addr 172.21.15.115:0/180210503 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:55.655 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f26a2ffd700 1 -- 172.21.15.115:0/180210503 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26980095f0 con 0x7f26a40ff880 2023-07-14T18:04:55.655 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f26a2ffd700 1 --2- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 0x7f26a419cab0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2698004f40 tx=0x7f2698005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:55.656 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f268bfff700 1 -- 172.21.15.115:0/180210503 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f269801d070 con 0x7f26a40ff880 2023-07-14T18:04:55.656 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f268bfff700 1 -- 172.21.15.115:0/180210503 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f26980044b0 con 0x7f26a40ff880 2023-07-14T18:04:55.656 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f268bfff700 1 -- 172.21.15.115:0/180210503 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2698003860 con 0x7f26a40ff880 2023-07-14T18:04:55.656 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a419d1f0 con 0x7f26a40ff880 2023-07-14T18:04:55.656 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.623+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a419d630 con 0x7f26a40ff880 2023-07-14T18:04:55.656 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.624+0000 7f268bfff700 1 -- 172.21.15.115:0/180210503 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f2698003a90 con 0x7f26a40ff880 2023-07-14T18:04:55.656 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.624+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a4061c70 con 0x7f26a40ff880 2023-07-14T18:04:55.657 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.624+0000 7f268bfff700 1 --2- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f268c038800 0x7f268c03acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:55.657 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.624+0000 7f268bfff700 1 -- 172.21.15.115:0/180210503 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f269804c600 con 0x7f26a40ff880 2023-07-14T18:04:55.657 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.624+0000 7f26a27fc700 1 --2- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f268c038800 0x7f268c03acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:55.657 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.625+0000 7f26a27fc700 1 --2- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f268c038800 0x7f268c03acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2694006fd0 tx=0x7f2694006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:55.658 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.629+0000 7f268bfff700 1 -- 172.21.15.115:0/180210503 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f269802a540 con 0x7f26a40ff880 2023-07-14T18:04:55.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.819+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26a4104cd0 con 0x7f268c038800 2023-07-14T18:04:55.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.820+0000 7f268bfff700 1 -- 172.21.15.115:0/180210503 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7f26a4104cd0 con 0x7f268c038800 2023-07-14T18:04:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:04:55.851 INFO:teuthology.orchestra.run.smithi115.stdout:[{"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}, {"addr": "172.21.15.187", "hostname": "smithi187", "labels": [], "status": ""}] 2023-07-14T18:04:55.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f268c038800 msgr2=0x7f268c03acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:55.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 --2- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f268c038800 0x7f268c03acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2694006fd0 tx=0x7f2694006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:55.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 msgr2=0x7f26a419cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:55.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 --2- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 0x7f26a419cab0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2698004f40 tx=0x7f2698005020 comp rx=0 tx=0).stop 2023-07-14T18:04:55.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 shutdown_connections 2023-07-14T18:04:55.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 --2- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f268c038800 0x7f268c03acb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:55.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 --2- 172.21.15.115:0/180210503 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f26a40ff880 0x7f26a419cab0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:55.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 >> 172.21.15.115:0/180210503 conn(0x7f26a4074e40 msgr2=0x7f26a40fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:55.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 shutdown_connections 2023-07-14T18:04:55.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:55.823+0000 7f26a92f6700 1 -- 172.21.15.115:0/180210503 wait complete. 2023-07-14T18:04:56.208 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:56 smithi115 ceph-mon[127451]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:04:56.208 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:56 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:56.278 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-14T18:04:56.279 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd crush tunables default 2023-07-14T18:04:56.827 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:04:57.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:57 smithi115 ceph-mon[127451]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:04:57.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.878+0000 7fe2a2620700 1 -- 172.21.15.115:0/2370074737 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 msgr2=0x7fe29c0fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:57.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.878+0000 7fe2a2620700 1 --2- 172.21.15.115:0/2370074737 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 0x7fe29c0fe950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe290009910 tx=0x7fe290009c20 comp rx=0 tx=0).stop 2023-07-14T18:04:57.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.878+0000 7fe2a2620700 1 -- 172.21.15.115:0/2370074737 shutdown_connections 2023-07-14T18:04:57.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.878+0000 7fe2a2620700 1 --2- 172.21.15.115:0/2370074737 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 0x7fe29c0fe950 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:57.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.878+0000 7fe2a2620700 1 -- 172.21.15.115:0/2370074737 >> 172.21.15.115:0/2370074737 conn(0x7fe29c075070 msgr2=0x7fe29c075470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:57.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.878+0000 7fe2a2620700 1 -- 172.21.15.115:0/2370074737 shutdown_connections 2023-07-14T18:04:57.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.878+0000 7fe2a2620700 1 -- 172.21.15.115:0/2370074737 wait complete. 2023-07-14T18:04:57.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.878+0000 7fe2a2620700 1 Processor -- start 2023-07-14T18:04:57.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.878+0000 7fe2a2620700 1 -- start start 2023-07-14T18:04:57.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe2a2620700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 0x7fe29c0ff500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:57.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe2a2620700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c101480 con 0x7fe29c0fe580 2023-07-14T18:04:57.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe29bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 0x7fe29c0ff500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:57.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe29bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 0x7fe29c0ff500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51636/0 (socket says 172.21.15.115:51636) 2023-07-14T18:04:57.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe29bfff700 1 -- 172.21.15.115:0/2073470833 learned_addr learned my addr 172.21.15.115:0/2073470833 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:04:57.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe29bfff700 1 -- 172.21.15.115:0/2073470833 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2900095f0 con 0x7fe29c0fe580 2023-07-14T18:04:57.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe29bfff700 1 --2- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 0x7fe29c0ff500 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe290006010 tx=0x7fe290004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:57.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe2997fa700 1 -- 172.21.15.115:0/2073470833 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe29001d070 con 0x7fe29c0fe580 2023-07-14T18:04:57.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe2997fa700 1 -- 172.21.15.115:0/2073470833 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe29000bdf0 con 0x7fe29c0fe580 2023-07-14T18:04:57.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe2997fa700 1 -- 172.21.15.115:0/2073470833 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe290022d80 con 0x7fe29c0fe580 2023-07-14T18:04:57.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c0ffa40 con 0x7fe29c0fe580 2023-07-14T18:04:57.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.879+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c0fff40 con 0x7fe29c0fe580 2023-07-14T18:04:57.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.880+0000 7fe2997fa700 1 -- 172.21.15.115:0/2073470833 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fe29000f460 con 0x7fe29c0fe580 2023-07-14T18:04:57.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.880+0000 7fe2997fa700 1 --2- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe284038750 0x7fe28403ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:04:57.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.880+0000 7fe2997fa700 1 -- 172.21.15.115:0/2073470833 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe29004caa0 con 0x7fe29c0fe580 2023-07-14T18:04:57.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.880+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c106da0 con 0x7fe29c0fe580 2023-07-14T18:04:57.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.881+0000 7fe29b7fe700 1 --2- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe284038750 0x7fe28403ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:04:57.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.881+0000 7fe29b7fe700 1 --2- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe284038750 0x7fe28403ac00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe28c006fd0 tx=0x7fe28c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:04:57.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:57.885+0000 7fe2997fa700 1 -- 172.21.15.115:0/2073470833 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe290027070 con 0x7fe29c0fe580 2023-07-14T18:04:58.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:58.080+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe29c061c70 con 0x7fe29c0fe580 2023-07-14T18:04:58.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:58 smithi115 ceph-mon[127451]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:04:59.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.033+0000 7fe2997fa700 1 -- 172.21.15.115:0/2073470833 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe29004a090 con 0x7fe29c0fe580 2023-07-14T18:04:59.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe284038750 msgr2=0x7fe28403ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 --2- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe284038750 0x7fe28403ac00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe28c006fd0 tx=0x7fe28c006e40 comp rx=0 tx=0).stop 2023-07-14T18:04:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 msgr2=0x7fe29c0ff500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:04:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 --2- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 0x7fe29c0ff500 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe290006010 tx=0x7fe290004f40 comp rx=0 tx=0).stop 2023-07-14T18:04:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 shutdown_connections 2023-07-14T18:04:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 --2- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe284038750 0x7fe28403ac00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:59.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 --2- 172.21.15.115:0/2073470833 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe29c0fe580 0x7fe29c0ff500 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:04:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 >> 172.21.15.115:0/2073470833 conn(0x7fe29c075070 msgr2=0x7fe29c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:04:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 shutdown_connections 2023-07-14T18:04:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:04:59.035+0000 7fe2a2620700 1 -- 172.21.15.115:0/2073470833 wait complete. 2023-07-14T18:04:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:adjusted tunables profile to default 2023-07-14T18:04:59.346 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:59 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2073470833' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T18:04:59.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:59.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:59.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:59.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:04:59.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:04:59.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:04:59.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:04:59.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:04:59 smithi115 ceph-mon[127451]: Updating smithi187:/etc/ceph/ceph.conf 2023-07-14T18:04:59.485 INFO:tasks.cephadm:Adding mon.a on smithi115 2023-07-14T18:04:59.485 INFO:tasks.cephadm:Adding mon.b on smithi150 2023-07-14T18:04:59.486 INFO:tasks.cephadm:Adding mon.c on smithi187 2023-07-14T18:04:59.486 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch apply mon '3;smithi115:172.21.15.115=a;smithi150:172.21.15.150=b;smithi187:172.21.15.187=c' 2023-07-14T18:05:00.059 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:00.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:00 smithi115 ceph-mon[127451]: Updating smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:00.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:00 smithi115 ceph-mon[127451]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:05:00.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:00 smithi115 ceph-mon[127451]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:00.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:00 smithi115 ceph-mon[127451]: Updating smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:05:00.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:00 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2073470833' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T18:05:00.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:00 smithi115 ceph-mon[127451]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:05:00.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:00 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:00.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:00 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:00.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:00 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:01.150 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.119+0000 7f86a8827700 1 -- 172.21.15.187:0/4155020162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 msgr2=0x7f86a4100bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:01.150 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.119+0000 7f86a8827700 1 --2- 172.21.15.187:0/4155020162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 0x7f86a4100bf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8698009910 tx=0x7f8698009c20 comp rx=0 tx=0).stop 2023-07-14T18:05:01.151 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.119+0000 7f86a8827700 1 -- 172.21.15.187:0/4155020162 shutdown_connections 2023-07-14T18:05:01.151 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.119+0000 7f86a8827700 1 --2- 172.21.15.187:0/4155020162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 0x7f86a4100bf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:01.151 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.119+0000 7f86a8827700 1 -- 172.21.15.187:0/4155020162 >> 172.21.15.187:0/4155020162 conn(0x7f86a40743e0 msgr2=0x7f86a40747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:01.151 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.120+0000 7f86a8827700 1 -- 172.21.15.187:0/4155020162 shutdown_connections 2023-07-14T18:05:01.151 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.120+0000 7f86a8827700 1 -- 172.21.15.187:0/4155020162 wait complete. 2023-07-14T18:05:01.151 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.120+0000 7f86a8827700 1 Processor -- start 2023-07-14T18:05:01.151 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.120+0000 7f86a8827700 1 -- start start 2023-07-14T18:05:01.151 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.120+0000 7f86a8827700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 0x7f86a419a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:01.152 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.120+0000 7f86a8827700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a419af30 con 0x7f86a4100820 2023-07-14T18:05:01.153 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86a259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 0x7f86a419a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:01.153 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86a259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 0x7f86a419a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.187:33468/0 (socket says 172.21.15.187:33468) 2023-07-14T18:05:01.153 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86a259c700 1 -- 172.21.15.187:0/2555562626 learned_addr learned my addr 172.21.15.187:0/2555562626 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:05:01.153 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86a259c700 1 -- 172.21.15.187:0/2555562626 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86980095f0 con 0x7f86a4100820 2023-07-14T18:05:01.154 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86a259c700 1 --2- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 0x7f86a419a9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8698004e90 tx=0x7f8698004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:01.154 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86937fe700 1 -- 172.21.15.187:0/2555562626 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f869801d070 con 0x7f86a4100820 2023-07-14T18:05:01.154 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a419b130 con 0x7f86a4100820 2023-07-14T18:05:01.154 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86937fe700 1 -- 172.21.15.187:0/2555562626 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f86980044b0 con 0x7f86a4100820 2023-07-14T18:05:01.154 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86937fe700 1 -- 172.21.15.187:0/2555562626 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86980038f0 con 0x7f86a4100820 2023-07-14T18:05:01.154 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.121+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a419b4f0 con 0x7f86a4100820 2023-07-14T18:05:01.154 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.122+0000 7f86937fe700 1 -- 172.21.15.187:0/2555562626 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f8698003b20 con 0x7f86a4100820 2023-07-14T18:05:01.155 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.122+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a4061c70 con 0x7f86a4100820 2023-07-14T18:05:01.155 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.122+0000 7f86937fe700 1 --2- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f868c038800 0x7f868c03acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:01.155 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.122+0000 7f86937fe700 1 -- 172.21.15.187:0/2555562626 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f869804c760 con 0x7f86a4100820 2023-07-14T18:05:01.155 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.123+0000 7f86a1d9b700 1 --2- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f868c038800 0x7f868c03acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:01.155 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.123+0000 7f86a1d9b700 1 --2- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f868c038800 0x7f868c03acb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8694006fd0 tx=0x7f8694006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:01.157 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.128+0000 7f86937fe700 1 -- 172.21.15.187:0/2555562626 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f869802a540 con 0x7f86a4100820 2023-07-14T18:05:01.350 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.320+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi115:172.21.15.115=a;smithi150:172.21.15.150=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f86a4106b30 con 0x7f868c038800 2023-07-14T18:05:01.359 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.330+0000 7f86937fe700 1 -- 172.21.15.187:0/2555562626 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f86a4106b30 con 0x7f868c038800 2023-07-14T18:05:01.360 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mon update... 2023-07-14T18:05:01.361 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.331+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f868c038800 msgr2=0x7f868c03acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:01.362 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.331+0000 7f86a8827700 1 --2- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f868c038800 0x7f868c03acb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8694006fd0 tx=0x7f8694006e40 comp rx=0 tx=0).stop 2023-07-14T18:05:01.362 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.332+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 msgr2=0x7f86a419a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:01.362 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.332+0000 7f86a8827700 1 --2- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 0x7f86a419a9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8698004e90 tx=0x7f8698004f70 comp rx=0 tx=0).stop 2023-07-14T18:05:01.362 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.332+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 shutdown_connections 2023-07-14T18:05:01.362 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.332+0000 7f86a8827700 1 --2- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f868c038800 0x7f868c03acb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:01.362 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.332+0000 7f86a8827700 1 --2- 172.21.15.187:0/2555562626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86a4100820 0x7f86a419a9f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:01.362 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.332+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 >> 172.21.15.187:0/2555562626 conn(0x7f86a40743e0 msgr2=0x7f86a41040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:01.363 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.332+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 shutdown_connections 2023-07-14T18:05:01.363 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:01.332+0000 7f86a8827700 1 -- 172.21.15.187:0/2555562626 wait complete. 2023-07-14T18:05:01.716 DEBUG:teuthology.orchestra.run.smithi150:mon.b> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.b.service 2023-07-14T18:05:01.718 DEBUG:teuthology.orchestra.run.smithi187:mon.c> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.c.service 2023-07-14T18:05:01.722 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T18:05:01.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T18:05:01.752 INFO:journalctl@ceph.mon.c.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:05:01.754 INFO:journalctl@ceph.mon.b.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:05:02.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:02 smithi115 ceph-mon[127451]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:02.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:02 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:02.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:02 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:02.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:02 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:02.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:02 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:02.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:02 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:02.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:02 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:02.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:02 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:02.356 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:03.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:03 smithi115 ceph-mon[127451]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi115:172.21.15.115=a;smithi150:172.21.15.150=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:03.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:03 smithi115 ceph-mon[127451]: Saving service mon spec with placement smithi115:172.21.15.115=a;smithi150:172.21.15.150=b;smithi187:172.21.15.187=c;count:3 2023-07-14T18:05:03.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:03 smithi115 ceph-mon[127451]: Deploying daemon mon.c on smithi187 2023-07-14T18:05:04.139 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 systemd[1]: Starting Ceph mon.c for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:05:04.297 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.266+0000 7fb25719a700 1 -- 172.21.15.187:0/2601240344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 msgr2=0x7fb2500fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:04.297 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.266+0000 7fb25719a700 1 --2- 172.21.15.187:0/2601240344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 0x7fb2500fe950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb244009910 tx=0x7fb244009c20 comp rx=0 tx=0).stop 2023-07-14T18:05:04.297 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.266+0000 7fb25719a700 1 -- 172.21.15.187:0/2601240344 shutdown_connections 2023-07-14T18:05:04.297 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.266+0000 7fb25719a700 1 --2- 172.21.15.187:0/2601240344 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 0x7fb2500fe950 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:04.298 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.266+0000 7fb25719a700 1 -- 172.21.15.187:0/2601240344 >> 172.21.15.187:0/2601240344 conn(0x7fb250075070 msgr2=0x7fb250075470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:04.298 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.266+0000 7fb25719a700 1 -- 172.21.15.187:0/2601240344 shutdown_connections 2023-07-14T18:05:04.298 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.266+0000 7fb25719a700 1 -- 172.21.15.187:0/2601240344 wait complete. 2023-07-14T18:05:04.298 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.266+0000 7fb25719a700 1 Processor -- start 2023-07-14T18:05:04.298 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.266+0000 7fb25719a700 1 -- start start 2023-07-14T18:05:04.298 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb25719a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 0x7fb25019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:04.298 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb25719a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25019ae50 con 0x7fb2500fe580 2023-07-14T18:05:04.299 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb254f36700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 0x7fb25019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:04.299 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb254f36700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 0x7fb25019a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.187:33474/0 (socket says 172.21.15.187:33474) 2023-07-14T18:05:04.299 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb254f36700 1 -- 172.21.15.187:0/2833946173 learned_addr learned my addr 172.21.15.187:0/2833946173 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:05:04.299 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb254f36700 1 -- 172.21.15.187:0/2833946173 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2440095f0 con 0x7fb2500fe580 2023-07-14T18:05:04.299 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb254f36700 1 --2- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 0x7fb25019a910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb244006010 tx=0x7fb244004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:04.300 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb24dffb700 1 -- 172.21.15.187:0/2833946173 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb24401d070 con 0x7fb2500fe580 2023-07-14T18:05:04.300 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25019b050 con 0x7fb2500fe580 2023-07-14T18:05:04.300 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.267+0000 7fb24dffb700 1 -- 172.21.15.187:0/2833946173 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb24400bdf0 con 0x7fb2500fe580 2023-07-14T18:05:04.301 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.268+0000 7fb24dffb700 1 -- 172.21.15.187:0/2833946173 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb244022d80 con 0x7fb2500fe580 2023-07-14T18:05:04.301 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.268+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25019b470 con 0x7fb2500fe580 2023-07-14T18:05:04.301 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.269+0000 7fb24dffb700 1 -- 172.21.15.187:0/2833946173 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fb24400f460 con 0x7fb2500fe580 2023-07-14T18:05:04.301 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.269+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb250061c70 con 0x7fb2500fe580 2023-07-14T18:05:04.301 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.269+0000 7fb24dffb700 1 --2- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb240038750 0x7fb24003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:04.301 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.269+0000 7fb24dffb700 1 -- 172.21.15.187:0/2833946173 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb24404ca60 con 0x7fb2500fe580 2023-07-14T18:05:04.301 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.269+0000 7fb24ffff700 1 --2- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb240038750 0x7fb24003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:04.302 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.270+0000 7fb24ffff700 1 --2- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb240038750 0x7fb24003ac00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb23c006fd0 tx=0x7fb23c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:04.304 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.274+0000 7fb24dffb700 1 -- 172.21.15.187:0/2833946173 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb244017440 con 0x7fb2500fe580 2023-07-14T18:05:04.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:04 smithi115 ceph-mon[127451]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:04.599 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.568+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb250195ea0 con 0x7fb2500fe580 2023-07-14T18:05:04.600 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.569+0000 7fb24dffb700 1 -- 172.21.15.187:0/2833946173 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+740 (secure 0 0 0) 0x7fb244027030 con 0x7fb2500fe580 2023-07-14T18:05:04.600 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T18:05:04.600 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","modified":"2023-07-14T18:03:35.805172Z","created":"2023-07-14T18:03:35.805172Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-14T18:05:04.601 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb240038750 msgr2=0x7fb24003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:04.602 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 --2- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb240038750 0x7fb24003ac00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb23c006fd0 tx=0x7fb23c006e40 comp rx=0 tx=0).stop 2023-07-14T18:05:04.602 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 msgr2=0x7fb25019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:04.602 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 --2- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 0x7fb25019a910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb244006010 tx=0x7fb244004f40 comp rx=0 tx=0).stop 2023-07-14T18:05:04.602 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 shutdown_connections 2023-07-14T18:05:04.602 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 --2- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb240038750 0x7fb24003ac00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:04.602 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 --2- 172.21.15.187:0/2833946173 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2500fe580 0x7fb25019a910 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:04.603 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 >> 172.21.15.187:0/2833946173 conn(0x7fb250075070 msgr2=0x7fb250103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:04.603 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 shutdown_connections 2023-07-14T18:05:04.603 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:04.572+0000 7fb25719a700 1 -- 172.21.15.187:0/2833946173 wait complete. 2023-07-14T18:05:04.603 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-07-14T18:05:04.734 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 podman[129274]: 2023-07-14T18:05:04.734 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T18:05:04.734 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T18:05:04.734 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: pidfile_write: ignore empty --pid-file 2023-07-14T18:05:04.734 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: load: jerasure load: lrc 2023-07-14T18:05:04.735 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: RocksDB version: 7.9.2 2023-07-14T18:05:04.735 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Git sha 0 2023-07-14T18:05:04.735 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T18:05:04.735 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: DB SUMMARY 2023-07-14T18:05:04.735 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: DB Session ID: FSR2B4UJV9JB03TG2TOC 2023-07-14T18:05:04.735 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: CURRENT file: CURRENT 2023-07-14T18:05:04.735 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T18:05:04.735 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T18:05:04.736 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-14T18:05:04.736 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-14T18:05:04.736 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.error_if_exists: 0 2023-07-14T18:05:04.736 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.create_if_missing: 0 2023-07-14T18:05:04.736 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.paranoid_checks: 1 2023-07-14T18:05:04.736 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T18:05:04.736 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T18:05:04.737 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T18:05:04.737 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.env: 0x55adf3bdfc60 2023-07-14T18:05:04.737 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T18:05:04.737 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.info_log: 0x55adf48dd2a0 2023-07-14T18:05:04.737 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T18:05:04.737 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.statistics: (nil) 2023-07-14T18:05:04.737 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.use_fsync: 0 2023-07-14T18:05:04.737 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_log_file_size: 0 2023-07-14T18:05:04.738 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T18:05:04.738 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T18:05:04.738 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T18:05:04.738 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T18:05:04.738 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.allow_fallocate: 1 2023-07-14T18:05:04.738 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T18:05:04.738 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T18:05:04.738 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.use_direct_reads: 0 2023-07-14T18:05:04.739 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T18:05:04.739 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T18:05:04.739 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.db_log_dir: 2023-07-14T18:05:04.739 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.wal_dir: 2023-07-14T18:05:04.739 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T18:05:04.739 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T18:05:04.739 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T18:05:04.739 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T18:05:04.740 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T18:05:04.740 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T18:05:04.740 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T18:05:04.740 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T18:05:04.740 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.write_buffer_manager: 0x55adf47fe3c0 2023-07-14T18:05:04.740 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T18:05:04.740 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T18:05:04.740 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T18:05:04.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T18:05:04.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T18:05:04.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T18:05:04.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T18:05:04.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T18:05:04.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.unordered_write: 0 2023-07-14T18:05:04.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T18:05:04.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T18:05:04.742 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T18:05:04.742 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T18:05:04.742 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.row_cache: None 2023-07-14T18:05:04.742 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.wal_filter: None 2023-07-14T18:05:04.742 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T18:05:04.742 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T18:05:04.742 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.two_write_queues: 0 2023-07-14T18:05:04.742 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T18:05:04.743 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.wal_compression: 0 2023-07-14T18:05:04.743 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.atomic_flush: 0 2023-07-14T18:05:04.743 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T18:05:04.743 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T18:05:04.743 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T18:05:04.743 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.log_readahead_size: 0 2023-07-14T18:05:04.743 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T18:05:04.743 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T18:05:04.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T18:05:04.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T18:05:04.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T18:05:04.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T18:05:04.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T18:05:04.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_background_jobs: 2 2023-07-14T18:05:04.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_background_compactions: -1 2023-07-14T18:05:04.745 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_subcompactions: 1 2023-07-14T18:05:04.745 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T18:05:04.745 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T18:05:04.745 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T18:05:04.745 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T18:05:04.745 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T18:05:04.745 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T18:05:04.745 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T18:05:04.746 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T18:05:04.746 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_open_files: -1 2023-07-14T18:05:04.746 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T18:05:04.746 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T18:05:04.746 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T18:05:04.746 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T18:05:04.746 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_background_flushes: -1 2023-07-14T18:05:04.746 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Compression algorithms supported: 2023-07-14T18:05:04.747 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T18:05:04.747 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: kZSTD supported: 0 2023-07-14T18:05:04.747 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: kXpressCompression supported: 0 2023-07-14T18:05:04.747 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T18:05:04.747 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: kZlibCompression supported: 1 2023-07-14T18:05:04.747 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: kSnappyCompression supported: 1 2023-07-14T18:05:04.747 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: kLZ4Compression supported: 1 2023-07-14T18:05:04.747 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: kBZip2Compression supported: 0 2023-07-14T18:05:04.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T18:05:04.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T18:05:04.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-14T18:05:04.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T18:05:04.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T18:05:04.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.merge_operator: 2023-07-14T18:05:04.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_filter: None 2023-07-14T18:05:04.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T18:05:04.749 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T18:05:04.749 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T18:05:04.749 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T18:05:04.749 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55adf48dd3a0) 2023-07-14T18:05:04.749 INFO:journalctl@ceph.mon.c.smithi187.stdout: cache_index_and_filter_blocks: 1 2023-07-14T18:05:04.749 INFO:journalctl@ceph.mon.c.smithi187.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T18:05:04.749 INFO:journalctl@ceph.mon.c.smithi187.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T18:05:04.750 INFO:journalctl@ceph.mon.c.smithi187.stdout: pin_top_level_index_and_filter: 1 2023-07-14T18:05:04.750 INFO:journalctl@ceph.mon.c.smithi187.stdout: index_type: 0 2023-07-14T18:05:04.750 INFO:journalctl@ceph.mon.c.smithi187.stdout: data_block_index_type: 0 2023-07-14T18:05:04.750 INFO:journalctl@ceph.mon.c.smithi187.stdout: index_shortening: 1 2023-07-14T18:05:04.750 INFO:journalctl@ceph.mon.c.smithi187.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T18:05:04.750 INFO:journalctl@ceph.mon.c.smithi187.stdout: checksum: 4 2023-07-14T18:05:04.750 INFO:journalctl@ceph.mon.c.smithi187.stdout: no_block_cache: 0 2023-07-14T18:05:04.750 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache: 0x55adf489d1f0 2023-07-14T18:05:04.751 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache_name: BinnedLRUCache 2023-07-14T18:05:04.751 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache_options: 2023-07-14T18:05:04.751 INFO:journalctl@ceph.mon.c.smithi187.stdout: capacity : 536870912 2023-07-14T18:05:04.751 INFO:journalctl@ceph.mon.c.smithi187.stdout: num_shard_bits : 4 2023-07-14T18:05:04.751 INFO:journalctl@ceph.mon.c.smithi187.stdout: strict_capacity_limit : 0 2023-07-14T18:05:04.751 INFO:journalctl@ceph.mon.c.smithi187.stdout: high_pri_pool_ratio: 0.000 2023-07-14T18:05:04.751 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache_compressed: (nil) 2023-07-14T18:05:04.751 INFO:journalctl@ceph.mon.c.smithi187.stdout: persistent_cache: (nil) 2023-07-14T18:05:04.752 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_size: 4096 2023-07-14T18:05:04.752 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_size_deviation: 10 2023-07-14T18:05:04.752 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_restart_interval: 16 2023-07-14T18:05:04.752 INFO:journalctl@ceph.mon.c.smithi187.stdout: index_block_restart_interval: 1 2023-07-14T18:05:04.752 INFO:journalctl@ceph.mon.c.smithi187.stdout: metadata_block_size: 4096 2023-07-14T18:05:04.752 INFO:journalctl@ceph.mon.c.smithi187.stdout: partition_filters: 0 2023-07-14T18:05:04.752 INFO:journalctl@ceph.mon.c.smithi187.stdout: use_delta_encoding: 1 2023-07-14T18:05:04.752 INFO:journalctl@ceph.mon.c.smithi187.stdout: filter_policy: bloomfilter 2023-07-14T18:05:04.753 INFO:journalctl@ceph.mon.c.smithi187.stdout: whole_key_filtering: 1 2023-07-14T18:05:04.753 INFO:journalctl@ceph.mon.c.smithi187.stdout: verify_compression: 0 2023-07-14T18:05:04.753 INFO:journalctl@ceph.mon.c.smithi187.stdout: read_amp_bytes_per_bit: 0 2023-07-14T18:05:04.753 INFO:journalctl@ceph.mon.c.smithi187.stdout: format_version: 5 2023-07-14T18:05:04.753 INFO:journalctl@ceph.mon.c.smithi187.stdout: enable_index_compression: 1 2023-07-14T18:05:04.753 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_align: 0 2023-07-14T18:05:04.753 INFO:journalctl@ceph.mon.c.smithi187.stdout: max_auto_readahead_size: 262144 2023-07-14T18:05:04.753 INFO:journalctl@ceph.mon.c.smithi187.stdout: prepopulate_block_cache: 0 2023-07-14T18:05:04.754 INFO:journalctl@ceph.mon.c.smithi187.stdout: initial_auto_readahead_size: 8192 2023-07-14T18:05:04.754 INFO:journalctl@ceph.mon.c.smithi187.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T18:05:04.754 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T18:05:04.754 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T18:05:04.754 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression: NoCompression 2023-07-14T18:05:04.754 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T18:05:04.754 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T18:05:04.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T18:05:04.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.num_levels: 7 2023-07-14T18:05:04.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T18:05:04.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T18:05:04.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T18:05:04.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T18:05:04.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T18:05:04.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T18:05:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T18:05:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:05:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T18:05:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T18:05:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:05:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:05:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T18:05:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T18:05:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T18:05:04.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T18:05:04.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:05:04.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:05:04.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T18:05:04.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T18:05:04.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:05:04.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T18:05:04.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T18:05:04.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T18:05:04.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T18:05:04.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T18:05:04.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T18:05:04.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T18:05:04.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T18:05:04.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T18:05:04.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T18:05:04.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T18:05:04.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T18:05:04.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T18:05:04.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T18:05:04.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T18:05:04.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T18:05:04.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T18:05:04.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T18:05:04.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T18:05:04.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T18:05:04.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T18:05:04.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T18:05:04.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T18:05:04.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T18:05:04.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T18:05:04.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T18:05:04.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T18:05:04.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T18:05:04.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T18:05:04.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T18:05:04.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T18:05:04.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T18:05:04.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.table_properties_collectors: 2023-07-14T18:05:04.763 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.inplace_update_support: 0 2023-07-14T18:05:04.763 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T18:05:04.763 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T18:05:04.763 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T18:05:04.763 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T18:05:04.763 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.bloom_locality: 0 2023-07-14T18:05:04.763 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.max_successive_merges: 0 2023-07-14T18:05:04.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T18:05:04.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T18:05:04.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T18:05:04.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T18:05:04.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.ttl: 2592000 2023-07-14T18:05:04.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T18:05:04.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T18:05:04.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T18:05:04.765 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.enable_blob_files: false 2023-07-14T18:05:04.765 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.min_blob_size: 0 2023-07-14T18:05:04.765 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 bash[129274]: 8bb9a2af086a5e87b0ca37b829bcb9d497adc93494ace2d0f9933c0aebb66004 2023-07-14T18:05:04.765 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T18:05:04.765 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T18:05:04.765 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T18:05:04.765 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T18:05:04.765 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T18:05:04.766 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T18:05:04.766 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T18:05:04.766 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T18:05:04.766 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T18:05:04.766 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T18:05:04.766 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e14406f8-45ae-4cf9-8a7f-a45a7227774c 2023-07-14T18:05:04.766 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689357904683672, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T18:05:04.766 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T18:05:04.767 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689357904684100, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689357904, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e14406f8-45ae-4cf9-8a7f-a45a7227774c", "db_session_id": "FSR2B4UJV9JB03TG2TOC", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T18:05:04.767 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689357904684168, "job": 1, "event": "recovery_finished"} 2023-07-14T18:05:04.767 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T18:05:04.767 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T18:05:04.767 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55adf4936000 2023-07-14T18:05:04.767 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: DB pointer 0x55adf4922000 2023-07-14T18:05:04.767 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T18:05:04.768 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T18:05:04.768 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** DB Stats ** 2023-07-14T18:05:04.768 INFO:journalctl@ceph.mon.c.smithi187.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:05:04.768 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T18:05:04.768 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:05:04.768 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:05:04.768 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T18:05:04.769 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:05:04.769 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:05:04.769 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2023-07-14T18:05:04.769 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** Compaction Stats [default] ** 2023-07-14T18:05:04.769 INFO:journalctl@ceph.mon.c.smithi187.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:05:04.769 INFO:journalctl@ceph.mon.c.smithi187.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T18:05:04.770 INFO:journalctl@ceph.mon.c.smithi187.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:05:04.770 INFO:journalctl@ceph.mon.c.smithi187.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:05:04.770 INFO:journalctl@ceph.mon.c.smithi187.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:05:04.770 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2023-07-14T18:05:04.770 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** Compaction Stats [default] ** 2023-07-14T18:05:04.770 INFO:journalctl@ceph.mon.c.smithi187.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:05:04.770 INFO:journalctl@ceph.mon.c.smithi187.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T18:05:04.771 INFO:journalctl@ceph.mon.c.smithi187.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:05:04.771 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2023-07-14T18:05:04.771 INFO:journalctl@ceph.mon.c.smithi187.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T18:05:04.771 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2023-07-14T18:05:04.771 INFO:journalctl@ceph.mon.c.smithi187.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:05:04.771 INFO:journalctl@ceph.mon.c.smithi187.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T18:05:04.772 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T18:05:04.772 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T18:05:04.772 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T18:05:04.772 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T18:05:04.772 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:05:04.772 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:05:04.772 INFO:journalctl@ceph.mon.c.smithi187.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T18:05:04.773 INFO:journalctl@ceph.mon.c.smithi187.stdout: Block cache BinnedLRUCache@0x55adf489d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-14T18:05:04.773 INFO:journalctl@ceph.mon.c.smithi187.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T18:05:04.773 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2023-07-14T18:05:04.773 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T18:05:04.773 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-14T18:05:04.773 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: using public_addr v2:172.21.15.187:0/0 -> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] 2023-07-14T18:05:04.773 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: starting mon.c rank -1 at public addrs [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] at bind addrs [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:05:04.774 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(???) e0 preinit fsid a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:05:04.774 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).mds e1 new map 2023-07-14T18:05:04.774 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-14T18:05:04.774 INFO:journalctl@ceph.mon.c.smithi187.stdout: e1 2023-07-14T18:05:04.774 INFO:journalctl@ceph.mon.c.smithi187.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:05:04.774 INFO:journalctl@ceph.mon.c.smithi187.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:05:04.774 INFO:journalctl@ceph.mon.c.smithi187.stdout: legacy client fscid: -1 2023-07-14T18:05:04.774 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2023-07-14T18:05:04.775 INFO:journalctl@ceph.mon.c.smithi187.stdout: No filesystems configured 2023-07-14T18:05:04.775 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T18:05:04.775 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T18:05:04.775 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T18:05:04.775 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T18:05:04.775 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T18:05:04.775 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T18:05:05.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T18:05:05.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:05:05.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:05:05.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:05:05.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: Added host smithi187 2023-07-14T18:05:05.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:05.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.159 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:05:05.159 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:05.159 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2073470833' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T18:05:05.159 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.159 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.159 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.159 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.160 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:05:05.160 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:05.160 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:05.160 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: Updating smithi187:/etc/ceph/ceph.conf 2023-07-14T18:05:05.160 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: Updating smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:05.160 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:05:05.160 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:05.161 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: Updating smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:05:05.161 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2073470833' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T18:05:05.161 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:05:05.161 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.161 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.161 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.161 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:05.161 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.162 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:05.162 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:05.162 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:05.162 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:05.162 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:05.162 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:05.162 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi115:172.21.15.115=a;smithi150:172.21.15.150=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:05.162 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: Saving service mon spec with placement smithi115:172.21.15.115=a;smithi150:172.21.15.150=b;smithi187:172.21.15.187=c;count:3 2023-07-14T18:05:05.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: Deploying daemon mon.c on smithi187 2023-07-14T18:05:05.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:05.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T18:05:05.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-14T18:05:05.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta expand map: {default=false} 2023-07-14T18:05:05.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta from 'false' to 'false' 2023-07-14T18:05:05.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta expanded map: {default=false} 2023-07-14T18:05:05.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta expand map: {default=info} 2023-07-14T18:05:05.164 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta from 'info' to 'info' 2023-07-14T18:05:05.164 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta expanded map: {default=info} 2023-07-14T18:05:05.164 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta expand map: {default=daemon} 2023-07-14T18:05:05.164 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T18:05:05.164 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T18:05:05.164 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta expand map: {default=debug} 2023-07-14T18:05:05.164 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T18:05:05.164 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 ceph-mon[129317]: expand_channel_meta expanded map: {default=debug} 2023-07-14T18:05:05.165 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:04 smithi187 systemd[1]: Started Ceph mon.c for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:05:06.209 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T18:05:06.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T18:05:06.771 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:05:10.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:05:10.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: mon.a calling monitor election 2023-07-14T18:05:10.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:10.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: mon.c calling monitor election 2023-07-14T18:05:10.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:10.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:10.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:05:10.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: monmap e2: 2 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} 2023-07-14T18:05:10.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: fsmap 2023-07-14T18:05:10.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:05:10.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: mgrmap e13: x(active, since 40s) 2023-07-14T18:05:10.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: overall HEALTH_OK 2023-07-14T18:05:10.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:10.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:10.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:10.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:05:10.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: mon.a calling monitor election 2023-07-14T18:05:10.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:10.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: mon.c calling monitor election 2023-07-14T18:05:10.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:10.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:10.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:10.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:05:10.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: monmap e2: 2 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} 2023-07-14T18:05:10.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: fsmap 2023-07-14T18:05:10.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:05:10.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: mgrmap e13: x(active, since 40s) 2023-07-14T18:05:10.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: overall HEALTH_OK 2023-07-14T18:05:10.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:10.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:10.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:10.966 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.936+0000 7fab90361700 1 -- 172.21.15.187:0/3860365428 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab740035c0 msgr2=0x7fab74005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:10.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.936+0000 7fab90361700 1 --2- 172.21.15.187:0/3860365428 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab740035c0 0x7fab74005a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fab88100cd0 tx=0x7fab7800fe60 comp rx=0 tx=0).stop 2023-07-14T18:05:10.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.936+0000 7fab90361700 1 -- 172.21.15.187:0/3860365428 shutdown_connections 2023-07-14T18:05:10.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.936+0000 7fab90361700 1 --2- 172.21.15.187:0/3860365428 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab740035c0 0x7fab74005a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:10.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.936+0000 7fab90361700 1 -- 172.21.15.187:0/3860365428 >> 172.21.15.187:0/3860365428 conn(0x7fab880fb340 msgr2=0x7fab880fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:10.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.936+0000 7fab90361700 1 -- 172.21.15.187:0/3860365428 shutdown_connections 2023-07-14T18:05:10.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.936+0000 7fab90361700 1 -- 172.21.15.187:0/3860365428 wait complete. 2023-07-14T18:05:10.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.937+0000 7fab90361700 1 Processor -- start 2023-07-14T18:05:10.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.937+0000 7fab90361700 1 -- start start 2023-07-14T18:05:10.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.937+0000 7fab90361700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab740035c0 0x7fab8819a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:10.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.937+0000 7fab90361700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 0x7fab88194a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:10.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.937+0000 7fab90361700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8819b430 con 0x7fab740035c0 2023-07-14T18:05:10.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.937+0000 7fab90361700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8819b5a0 con 0x7fab8819af00 2023-07-14T18:05:10.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8d8fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 0x7fab88194a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:10.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8d8fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 0x7fab88194a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:58412/0 (socket says 172.21.15.187:58412) 2023-07-14T18:05:10.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8d8fc700 1 -- 172.21.15.187:0/2082339242 learned_addr learned my addr 172.21.15.187:0/2082339242 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:05:10.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8e0fd700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab740035c0 0x7fab8819a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:10.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8d8fc700 1 -- 172.21.15.187:0/2082339242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 msgr2=0x7fab88194a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T18:05:10.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8d8fc700 1 -- 172.21.15.187:0/2082339242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 msgr2=0x7fab88194a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T18:05:10.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8d8fc700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 0x7fab88194a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T18:05:10.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8d8fc700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 0x7fab88194a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:05:10.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8e0fd700 1 -- 172.21.15.187:0/2082339242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 msgr2=0x7fab88194a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:05:10.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8e0fd700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 0x7fab88194a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:10.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8e0fd700 1 -- 172.21.15.187:0/2082339242 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7800f8b0 con 0x7fab740035c0 2023-07-14T18:05:10.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab8e0fd700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab740035c0 0x7fab8819a9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fab7801b000 tx=0x7fab78013c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:10.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab7f7fe700 1 -- 172.21.15.187:0/2082339242 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fab78023070 con 0x7fab740035c0 2023-07-14T18:05:10.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.938+0000 7fab7f7fe700 1 -- 172.21.15.187:0/2082339242 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fab780037c0 con 0x7fab740035c0 2023-07-14T18:05:10.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.939+0000 7fab7f7fe700 1 -- 172.21.15.187:0/2082339242 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fab7801cb70 con 0x7fab740035c0 2023-07-14T18:05:10.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.939+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab88195040 con 0x7fab740035c0 2023-07-14T18:05:10.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.939+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab88195610 con 0x7fab740035c0 2023-07-14T18:05:10.973 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.940+0000 7fab7f7fe700 1 -- 172.21.15.187:0/2082339242 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fab78031440 con 0x7fab740035c0 2023-07-14T18:05:10.973 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.940+0000 7fab7f7fe700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fab6c0385f0 0x7fab6c03aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:10.973 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.940+0000 7fab7f7fe700 1 -- 172.21.15.187:0/2082339242 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fab7804d410 con 0x7fab740035c0 2023-07-14T18:05:10.973 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.940+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab88061c70 con 0x7fab740035c0 2023-07-14T18:05:10.973 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.940+0000 7fab8d8fc700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fab6c0385f0 0x7fab6c03aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:10.973 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.941+0000 7fab8d8fc700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fab6c0385f0 0x7fab6c03aaa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab84009180 tx=0x7fab84008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:10.975 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:10.945+0000 7fab7f7fe700 1 -- 172.21.15.187:0/2082339242 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fab7801cde0 con 0x7fab740035c0 2023-07-14T18:05:11.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:11.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:11.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:10 smithi187 ceph-mon[129317]: Deploying daemon mon.b on smithi150 2023-07-14T18:05:11.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:11.238 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.208+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab8819f7d0 con 0x7fab740035c0 2023-07-14T18:05:11.238 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.208+0000 7fab7f7fe700 1 -- 172.21.15.187:0/2082339242 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1010 (secure 0 0 0) 0x7fab7804a020 con 0x7fab740035c0 2023-07-14T18:05:11.239 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T18:05:11.239 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","modified":"2023-07-14T18:05:04.717538Z","created":"2023-07-14T18:03:35.805172Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T18:05:11.241 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fab6c0385f0 msgr2=0x7fab6c03aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:11.241 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fab6c0385f0 0x7fab6c03aaa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab84009180 tx=0x7fab84008040 comp rx=0 tx=0).stop 2023-07-14T18:05:11.241 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab740035c0 msgr2=0x7fab8819a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:11.241 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab740035c0 0x7fab8819a9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fab7801b000 tx=0x7fab78013c40 comp rx=0 tx=0).stop 2023-07-14T18:05:11.241 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 shutdown_connections 2023-07-14T18:05:11.242 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fab6c0385f0 0x7fab6c03aaa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:11.242 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fab740035c0 0x7fab8819a9c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:11.242 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 --2- 172.21.15.187:0/2082339242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fab8819af00 0x7fab88194a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:11.242 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 >> 172.21.15.187:0/2082339242 conn(0x7fab880fb340 msgr2=0x7fab8818dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:11.242 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 shutdown_connections 2023-07-14T18:05:11.242 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:11.211+0000 7fab90361700 1 -- 172.21.15.187:0/2082339242 wait complete. 2023-07-14T18:05:11.243 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2023-07-14T18:05:11.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:11.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:11.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:10 smithi115 ceph-mon[127451]: Deploying daemon mon.b on smithi150 2023-07-14T18:05:11.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:12.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:11 smithi187 ceph-mon[129317]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:11 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/2082339242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:05:12.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:11 smithi115 ceph-mon[127451]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:11 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/2082339242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:05:12.594 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 systemd[1]: Starting Ceph mon.b for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:05:12.623 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T18:05:12.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T18:05:12.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 podman[128813]: 2023-07-14T18:05:12.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 bash[128813]: 8bad80ddd87636214351606c3702035bb9ce3612fcaaf13f653e18c4543deafc 2023-07-14T18:05:12.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T18:05:12.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T18:05:12.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pidfile_write: ignore empty --pid-file 2023-07-14T18:05:12.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: load: jerasure load: lrc 2023-07-14T18:05:12.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: RocksDB version: 7.9.2 2023-07-14T18:05:12.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Git sha 0 2023-07-14T18:05:12.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T18:05:12.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: DB SUMMARY 2023-07-14T18:05:12.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: DB Session ID: TTHYZ8A2V536UBW5HVAO 2023-07-14T18:05:12.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: CURRENT file: CURRENT 2023-07-14T18:05:12.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T18:05:12.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T18:05:12.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-14T18:05:12.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-14T18:05:12.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.error_if_exists: 0 2023-07-14T18:05:12.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.create_if_missing: 0 2023-07-14T18:05:12.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.paranoid_checks: 1 2023-07-14T18:05:12.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T18:05:12.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T18:05:12.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T18:05:12.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.env: 0x5566861d2c60 2023-07-14T18:05:12.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T18:05:12.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.info_log: 0x55668890f2a0 2023-07-14T18:05:12.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T18:05:12.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.statistics: (nil) 2023-07-14T18:05:12.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.use_fsync: 0 2023-07-14T18:05:12.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_log_file_size: 0 2023-07-14T18:05:12.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T18:05:12.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T18:05:12.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T18:05:12.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T18:05:12.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.allow_fallocate: 1 2023-07-14T18:05:12.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T18:05:12.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T18:05:12.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.use_direct_reads: 0 2023-07-14T18:05:12.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T18:05:12.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T18:05:12.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.db_log_dir: 2023-07-14T18:05:12.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.wal_dir: 2023-07-14T18:05:12.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T18:05:12.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T18:05:12.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T18:05:12.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T18:05:12.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T18:05:12.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T18:05:12.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T18:05:12.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T18:05:12.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.write_buffer_manager: 0x5566888303c0 2023-07-14T18:05:12.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T18:05:12.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T18:05:12.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T18:05:12.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T18:05:12.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T18:05:12.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T18:05:12.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T18:05:12.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T18:05:12.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.unordered_write: 0 2023-07-14T18:05:12.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T18:05:12.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T18:05:12.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T18:05:12.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T18:05:12.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.row_cache: None 2023-07-14T18:05:12.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.wal_filter: None 2023-07-14T18:05:12.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T18:05:12.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T18:05:12.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.two_write_queues: 0 2023-07-14T18:05:12.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T18:05:12.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.wal_compression: 0 2023-07-14T18:05:12.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.atomic_flush: 0 2023-07-14T18:05:12.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T18:05:12.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T18:05:12.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T18:05:12.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.log_readahead_size: 0 2023-07-14T18:05:12.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T18:05:12.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T18:05:12.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T18:05:12.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T18:05:12.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T18:05:12.874 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T18:05:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T18:05:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_background_jobs: 2 2023-07-14T18:05:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_background_compactions: -1 2023-07-14T18:05:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_subcompactions: 1 2023-07-14T18:05:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T18:05:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T18:05:12.875 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T18:05:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T18:05:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T18:05:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T18:05:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T18:05:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T18:05:12.876 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_open_files: -1 2023-07-14T18:05:12.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T18:05:12.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T18:05:12.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T18:05:12.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T18:05:12.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_background_flushes: -1 2023-07-14T18:05:12.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Compression algorithms supported: 2023-07-14T18:05:12.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T18:05:12.877 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: kZSTD supported: 0 2023-07-14T18:05:12.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: kXpressCompression supported: 0 2023-07-14T18:05:12.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T18:05:12.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: kZlibCompression supported: 1 2023-07-14T18:05:12.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: kSnappyCompression supported: 1 2023-07-14T18:05:12.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: kLZ4Compression supported: 1 2023-07-14T18:05:12.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: kBZip2Compression supported: 0 2023-07-14T18:05:12.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T18:05:12.878 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T18:05:12.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-14T18:05:12.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T18:05:12.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T18:05:12.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.merge_operator: 2023-07-14T18:05:12.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_filter: None 2023-07-14T18:05:12.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T18:05:12.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T18:05:12.879 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T18:05:12.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T18:05:12.880 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55668890f3a0) 2023-07-14T18:05:12.880 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks: 1 2023-07-14T18:05:12.880 INFO:journalctl@ceph.mon.b.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T18:05:12.880 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T18:05:12.880 INFO:journalctl@ceph.mon.b.smithi150.stdout: pin_top_level_index_and_filter: 1 2023-07-14T18:05:12.880 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_type: 0 2023-07-14T18:05:12.881 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_index_type: 0 2023-07-14T18:05:12.881 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_shortening: 1 2023-07-14T18:05:12.881 INFO:journalctl@ceph.mon.b.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T18:05:12.881 INFO:journalctl@ceph.mon.b.smithi150.stdout: checksum: 4 2023-07-14T18:05:12.881 INFO:journalctl@ceph.mon.b.smithi150.stdout: no_block_cache: 0 2023-07-14T18:05:12.881 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache: 0x5566888cf1f0 2023-07-14T18:05:12.881 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_name: BinnedLRUCache 2023-07-14T18:05:12.881 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_options: 2023-07-14T18:05:12.882 INFO:journalctl@ceph.mon.b.smithi150.stdout: capacity : 536870912 2023-07-14T18:05:12.882 INFO:journalctl@ceph.mon.b.smithi150.stdout: num_shard_bits : 4 2023-07-14T18:05:12.882 INFO:journalctl@ceph.mon.b.smithi150.stdout: strict_capacity_limit : 0 2023-07-14T18:05:12.882 INFO:journalctl@ceph.mon.b.smithi150.stdout: high_pri_pool_ratio: 0.000 2023-07-14T18:05:12.882 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_cache_compressed: (nil) 2023-07-14T18:05:12.882 INFO:journalctl@ceph.mon.b.smithi150.stdout: persistent_cache: (nil) 2023-07-14T18:05:12.882 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size: 4096 2023-07-14T18:05:12.882 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_size_deviation: 10 2023-07-14T18:05:12.883 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_restart_interval: 16 2023-07-14T18:05:12.883 INFO:journalctl@ceph.mon.b.smithi150.stdout: index_block_restart_interval: 1 2023-07-14T18:05:12.883 INFO:journalctl@ceph.mon.b.smithi150.stdout: metadata_block_size: 4096 2023-07-14T18:05:12.883 INFO:journalctl@ceph.mon.b.smithi150.stdout: partition_filters: 0 2023-07-14T18:05:12.883 INFO:journalctl@ceph.mon.b.smithi150.stdout: use_delta_encoding: 1 2023-07-14T18:05:12.883 INFO:journalctl@ceph.mon.b.smithi150.stdout: filter_policy: bloomfilter 2023-07-14T18:05:12.883 INFO:journalctl@ceph.mon.b.smithi150.stdout: whole_key_filtering: 1 2023-07-14T18:05:12.883 INFO:journalctl@ceph.mon.b.smithi150.stdout: verify_compression: 0 2023-07-14T18:05:12.884 INFO:journalctl@ceph.mon.b.smithi150.stdout: read_amp_bytes_per_bit: 0 2023-07-14T18:05:12.884 INFO:journalctl@ceph.mon.b.smithi150.stdout: format_version: 5 2023-07-14T18:05:12.884 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_index_compression: 1 2023-07-14T18:05:12.884 INFO:journalctl@ceph.mon.b.smithi150.stdout: block_align: 0 2023-07-14T18:05:12.884 INFO:journalctl@ceph.mon.b.smithi150.stdout: max_auto_readahead_size: 262144 2023-07-14T18:05:12.884 INFO:journalctl@ceph.mon.b.smithi150.stdout: prepopulate_block_cache: 0 2023-07-14T18:05:12.884 INFO:journalctl@ceph.mon.b.smithi150.stdout: initial_auto_readahead_size: 8192 2023-07-14T18:05:12.884 INFO:journalctl@ceph.mon.b.smithi150.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T18:05:12.885 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T18:05:12.885 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T18:05:12.885 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression: NoCompression 2023-07-14T18:05:12.885 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T18:05:12.885 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T18:05:12.885 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T18:05:12.885 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.num_levels: 7 2023-07-14T18:05:12.885 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T18:05:12.886 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T18:05:12.886 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T18:05:12.886 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T18:05:12.886 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T18:05:12.886 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T18:05:12.886 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T18:05:12.886 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:05:12.887 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T18:05:12.887 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T18:05:12.887 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:05:12.887 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:05:12.887 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T18:05:12.887 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T18:05:12.887 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T18:05:12.887 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T18:05:12.888 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:05:12.888 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:05:12.888 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T18:05:12.888 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T18:05:12.888 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:05:12.888 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T18:05:12.888 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T18:05:12.888 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T18:05:12.889 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T18:05:12.889 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T18:05:12.889 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T18:05:12.889 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T18:05:12.889 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T18:05:12.889 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T18:05:12.890 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T18:05:12.890 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T18:05:12.890 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T18:05:12.890 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T18:05:12.890 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T18:05:12.890 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T18:05:12.890 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T18:05:12.890 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T18:05:12.891 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T18:05:12.891 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T18:05:12.891 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T18:05:12.891 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T18:05:12.891 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T18:05:12.891 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T18:05:12.891 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T18:05:12.891 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T18:05:12.892 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T18:05:12.892 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T18:05:12.892 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T18:05:12.892 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T18:05:12.892 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T18:05:12.892 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T18:05:12.892 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T18:05:12.892 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.table_properties_collectors: 2023-07-14T18:05:12.893 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.inplace_update_support: 0 2023-07-14T18:05:12.893 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T18:05:12.893 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T18:05:12.893 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T18:05:12.893 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T18:05:12.893 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.bloom_locality: 0 2023-07-14T18:05:12.894 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.max_successive_merges: 0 2023-07-14T18:05:12.894 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T18:05:12.894 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T18:05:12.894 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T18:05:12.894 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T18:05:12.894 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.ttl: 2592000 2023-07-14T18:05:12.894 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T18:05:12.894 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T18:05:12.895 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T18:05:12.895 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.enable_blob_files: false 2023-07-14T18:05:12.895 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.min_blob_size: 0 2023-07-14T18:05:12.895 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T18:05:12.895 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T18:05:12.895 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T18:05:12.895 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T18:05:12.896 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T18:05:12.896 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T18:05:12.896 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T18:05:12.896 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T18:05:12.896 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T18:05:12.896 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T18:05:12.896 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f62ebeed-ef74-4221-bd99-9334e7df5d0b 2023-07-14T18:05:12.896 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689357912748461, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T18:05:12.897 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T18:05:12.897 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689357912749028, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689357912, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f62ebeed-ef74-4221-bd99-9334e7df5d0b", "db_session_id": "TTHYZ8A2V536UBW5HVAO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T18:05:12.897 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689357912749118, "job": 1, "event": "recovery_finished"} 2023-07-14T18:05:12.897 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T18:05:12.897 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T18:05:12.897 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556688968000 2023-07-14T18:05:12.897 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: DB pointer 0x556688954000 2023-07-14T18:05:12.898 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T18:05:12.898 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T18:05:12.898 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** DB Stats ** 2023-07-14T18:05:12.898 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:05:12.898 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T18:05:12.898 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:05:12.898 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:05:12.898 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T18:05:12.899 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:05:12.899 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:05:12.899 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2023-07-14T18:05:12.899 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2023-07-14T18:05:12.899 INFO:journalctl@ceph.mon.b.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:05:12.899 INFO:journalctl@ceph.mon.b.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T18:05:12.899 INFO:journalctl@ceph.mon.b.smithi150.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:05:12.899 INFO:journalctl@ceph.mon.b.smithi150.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:05:12.900 INFO:journalctl@ceph.mon.b.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:05:12.900 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2023-07-14T18:05:12.900 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** Compaction Stats [default] ** 2023-07-14T18:05:12.900 INFO:journalctl@ceph.mon.b.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:05:12.900 INFO:journalctl@ceph.mon.b.smithi150.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T18:05:12.900 INFO:journalctl@ceph.mon.b.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:05:12.900 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2023-07-14T18:05:12.900 INFO:journalctl@ceph.mon.b.smithi150.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T18:05:12.901 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2023-07-14T18:05:12.901 INFO:journalctl@ceph.mon.b.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:05:12.901 INFO:journalctl@ceph.mon.b.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T18:05:12.901 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T18:05:12.901 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T18:05:12.901 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T18:05:12.901 INFO:journalctl@ceph.mon.b.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T18:05:12.901 INFO:journalctl@ceph.mon.b.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:05:12.902 INFO:journalctl@ceph.mon.b.smithi150.stdout: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:05:12.902 INFO:journalctl@ceph.mon.b.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T18:05:12.902 INFO:journalctl@ceph.mon.b.smithi150.stdout: Block cache BinnedLRUCache@0x5566888cf1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-14T18:05:12.902 INFO:journalctl@ceph.mon.b.smithi150.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T18:05:12.902 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2023-07-14T18:05:12.902 INFO:journalctl@ceph.mon.b.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T18:05:12.902 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-14T18:05:12.902 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: using public_addr v2:172.21.15.150:0/0 -> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-07-14T18:05:12.903 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: starting mon.b rank -1 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:05:12.903 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(???) e0 preinit fsid a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:05:12.903 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 systemd[1]: Started Ceph mon.b for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:05:12.903 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).mds e1 new map 2023-07-14T18:05:12.903 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-14T18:05:12.903 INFO:journalctl@ceph.mon.b.smithi150.stdout: e1 2023-07-14T18:05:12.904 INFO:journalctl@ceph.mon.b.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:05:12.904 INFO:journalctl@ceph.mon.b.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:05:12.904 INFO:journalctl@ceph.mon.b.smithi150.stdout: legacy client fscid: -1 2023-07-14T18:05:12.904 INFO:journalctl@ceph.mon.b.smithi150.stdout: 2023-07-14T18:05:12.904 INFO:journalctl@ceph.mon.b.smithi150.stdout: No filesystems configured 2023-07-14T18:05:12.904 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T18:05:12.905 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T18:05:12.905 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T18:05:12.905 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T18:05:12.905 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T18:05:12.905 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T18:05:12.905 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T18:05:12.905 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:05:12.905 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:05:12.906 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:05:12.906 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: Added host smithi187 2023-07-14T18:05:12.906 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.906 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.906 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.906 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:05:12.906 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.906 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2073470833' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T18:05:12.907 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.907 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.907 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.907 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.907 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:05:12.907 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:12.907 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:12.907 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: Updating smithi187:/etc/ceph/ceph.conf 2023-07-14T18:05:12.908 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: Updating smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:12.908 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:05:12.908 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.908 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: Updating smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:05:12.908 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2073470833' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T18:05:12.908 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:05:12.908 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.909 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.909 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.909 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.909 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.909 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:12.909 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:12.909 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:12.909 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.910 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:12.910 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:12.910 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi115:172.21.15.115=a;smithi150:172.21.15.150=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:12.910 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: Saving service mon spec with placement smithi115:172.21.15.115=a;smithi150:172.21.15.150=b;smithi187:172.21.15.187=c;count:3 2023-07-14T18:05:12.910 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: Deploying daemon mon.c on smithi187 2023-07-14T18:05:12.910 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.910 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:05:12.910 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.a calling monitor election 2023-07-14T18:05:12.911 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:12.911 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.911 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:12.911 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:12.911 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.c calling monitor election 2023-07-14T18:05:12.911 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.911 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:12.911 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:12.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:12.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:05:12.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: monmap e2: 2 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} 2023-07-14T18:05:12.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: fsmap 2023-07-14T18:05:12.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:05:12.912 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mgrmap e13: x(active, since 40s) 2023-07-14T18:05:12.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: overall HEALTH_OK 2023-07-14T18:05:12.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:12.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:12.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:12.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: Deploying daemon mon.b on smithi150 2023-07-14T18:05:12.913 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:12.914 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:12.914 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/2082339242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:05:12.914 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T18:05:12.914 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:12 smithi150 ceph-mon[128853]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-14T18:05:13.185 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:05:17.964 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.933+0000 7fe60dd4a700 1 -- 172.21.15.187:0/2109782941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 msgr2=0x7fe5f0007820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:17.964 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.933+0000 7fe60dd4a700 1 --2- 172.21.15.187:0/2109782941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 0x7fe5f0007820 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc005fa0 tx=0x7fe5fc0104b0 comp rx=0 tx=0).stop 2023-07-14T18:05:17.964 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.934+0000 7fe60dd4a700 1 -- 172.21.15.187:0/2109782941 shutdown_connections 2023-07-14T18:05:17.965 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.934+0000 7fe60dd4a700 1 --2- 172.21.15.187:0/2109782941 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 0x7fe5f0007820 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:17.965 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.934+0000 7fe60dd4a700 1 -- 172.21.15.187:0/2109782941 >> 172.21.15.187:0/2109782941 conn(0x7fe608073ea0 msgr2=0x7fe6080742a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:17.965 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.934+0000 7fe60dd4a700 1 -- 172.21.15.187:0/2109782941 shutdown_connections 2023-07-14T18:05:17.965 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.934+0000 7fe60dd4a700 1 -- 172.21.15.187:0/2109782941 wait complete. 2023-07-14T18:05:17.966 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.935+0000 7fe60dd4a700 1 Processor -- start 2023-07-14T18:05:17.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.935+0000 7fe60dd4a700 1 -- start start 2023-07-14T18:05:17.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe60dd4a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 0x7fe6081988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:17.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe60dd4a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe608198e00 0x7fe608192940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:17.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe60dd4a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 0x7fe608193320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:17.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe60dd4a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe608199410 con 0x7fe608102f20 2023-07-14T18:05:17.967 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe60dd4a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60819f6e0 con 0x7fe608198e00 2023-07-14T18:05:17.968 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe60dd4a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60819f820 con 0x7fe608192e80 2023-07-14T18:05:17.968 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe607fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 0x7fe608193320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:17.968 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe6077fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 0x7fe6081988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:17.968 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe606ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe608198e00 0x7fe608192940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:17.968 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe607fff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 0x7fe608193320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:58438/0 (socket says 172.21.15.187:58438) 2023-07-14T18:05:17.968 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe6077fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 0x7fe6081988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.187:33530/0 (socket says 172.21.15.187:33530) 2023-07-14T18:05:17.968 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe607fff700 1 -- 172.21.15.187:0/1318728866 learned_addr learned my addr 172.21.15.187:0/1318728866 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:05:17.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe607fff700 1 -- 172.21.15.187:0/1318728866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 msgr2=0x7fe608193320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-14T18:05:17.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe607fff700 1 -- 172.21.15.187:0/1318728866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 msgr2=0x7fe608193320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T18:05:17.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe607fff700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 0x7fe608193320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T18:05:17.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe606ffd700 1 -- 172.21.15.187:0/1318728866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe608198e00 msgr2=0x7fe608192940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T18:05:17.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe606ffd700 1 -- 172.21.15.187:0/1318728866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe608198e00 msgr2=0x7fe608192940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T18:05:17.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe606ffd700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe608198e00 0x7fe608192940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T18:05:17.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe607fff700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 0x7fe608193320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:05:17.969 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe606ffd700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe608198e00 0x7fe608192940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:05:17.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe6077fe700 1 -- 172.21.15.187:0/1318728866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 msgr2=0x7fe608193320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:05:17.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe6077fe700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 0x7fe608193320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:17.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe6077fe700 1 -- 172.21.15.187:0/1318728866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe608198e00 msgr2=0x7fe608192940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:05:17.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe6077fe700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe608198e00 0x7fe608192940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:17.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe6077fe700 1 -- 172.21.15.187:0/1318728866 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc0149d0 con 0x7fe608102f20 2023-07-14T18:05:17.970 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.936+0000 7fe6077fe700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 0x7fe6081988c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc005fa0 tx=0x7fe5fc005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:17.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.937+0000 7fe604ff9700 1 -- 172.21.15.187:0/1318728866 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc019070 con 0x7fe608102f20 2023-07-14T18:05:17.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.937+0000 7fe604ff9700 1 -- 172.21.15.187:0/1318728866 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe5fc01a040 con 0x7fe608102f20 2023-07-14T18:05:17.971 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.937+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60819faa0 con 0x7fe608102f20 2023-07-14T18:05:17.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.937+0000 7fe604ff9700 1 -- 172.21.15.187:0/1318728866 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc01eb50 con 0x7fe608102f20 2023-07-14T18:05:17.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.937+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60819fff0 con 0x7fe608102f20 2023-07-14T18:05:17.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.938+0000 7fe604ff9700 1 -- 172.21.15.187:0/1318728866 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fe5fc01ecb0 con 0x7fe608102f20 2023-07-14T18:05:17.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.938+0000 7fe604ff9700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5f4038c80 0x7fe5f403b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:17.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.938+0000 7fe604ff9700 1 -- 172.21.15.187:0/1318728866 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe5fc04d9d0 con 0x7fe608102f20 2023-07-14T18:05:17.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.938+0000 7fe606ffd700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5f4038c80 0x7fe5f403b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:17.972 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.938+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe608061c70 con 0x7fe608102f20 2023-07-14T18:05:17.973 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.939+0000 7fe606ffd700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5f4038c80 0x7fe5f403b130 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009850 tx=0x7fe5f8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:17.973 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:17.943+0000 7fe604ff9700 1 -- 172.21.15.187:0/1318728866 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe5fc031430 con 0x7fe608102f20 2023-07-14T18:05:18.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:05:18.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: mon.a calling monitor election 2023-07-14T18:05:18.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: mon.c calling monitor election 2023-07-14T18:05:18.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:18.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:18.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: mon.b calling monitor election 2023-07-14T18:05:18.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:18.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:18.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T18:05:18.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} 2023-07-14T18:05:18.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: fsmap 2023-07-14T18:05:18.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:05:18.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: mgrmap e13: x(active, since 49s) 2023-07-14T18:05:18.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: overall HEALTH_OK 2023-07-14T18:05:18.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.261 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.230+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6081a01e0 con 0x7fe608102f20 2023-07-14T18:05:18.261 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.231+0000 7fe604ff9700 1 -- 172.21.15.187:0/1318728866 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1280 (secure 0 0 0) 0x7fe5fc026390 con 0x7fe608102f20 2023-07-14T18:05:18.262 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-07-14T18:05:18.262 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":3,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","modified":"2023-07-14T18:05:12.778091Z","created":"2023-07-14T18:03:35.805172Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-07-14T18:05:18.264 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.233+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5f4038c80 msgr2=0x7fe5f403b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:18.264 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5f4038c80 0x7fe5f403b130 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009850 tx=0x7fe5f8008040 comp rx=0 tx=0).stop 2023-07-14T18:05:18.264 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 msgr2=0x7fe6081988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:18.264 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 0x7fe6081988c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc005fa0 tx=0x7fe5fc005410 comp rx=0 tx=0).stop 2023-07-14T18:05:18.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 shutdown_connections 2023-07-14T18:05:18.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5f4038c80 0x7fe5f403b130 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:18.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe608192e80 0x7fe608193320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:18.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe608102f20 0x7fe6081988c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:18.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 --2- 172.21.15.187:0/1318728866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe608198e00 0x7fe608192940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:18.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 >> 172.21.15.187:0/1318728866 conn(0x7fe608073ea0 msgr2=0x7fe6080fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:18.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 shutdown_connections 2023-07-14T18:05:18.266 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:18.234+0000 7fe60dd4a700 1 -- 172.21.15.187:0/1318728866 wait complete. 2023-07-14T18:05:18.266 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 3 2023-07-14T18:05:18.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:05:18.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: mon.a calling monitor election 2023-07-14T18:05:18.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: mon.c calling monitor election 2023-07-14T18:05:18.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:18.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:18.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: mon.b calling monitor election 2023-07-14T18:05:18.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:18.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:18.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T18:05:18.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} 2023-07-14T18:05:18.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: fsmap 2023-07-14T18:05:18.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:05:18.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: mgrmap e13: x(active, since 49s) 2023-07-14T18:05:18.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: overall HEALTH_OK 2023-07-14T18:05:18.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:05:18.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: mon.a calling monitor election 2023-07-14T18:05:18.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: mon.c calling monitor election 2023-07-14T18:05:18.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:05:18.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:18.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: mon.b calling monitor election 2023-07-14T18:05:18.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:18.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:18.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:18.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T18:05:18.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0],c=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} 2023-07-14T18:05:18.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: fsmap 2023-07-14T18:05:18.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:05:18.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: mgrmap e13: x(active, since 49s) 2023-07-14T18:05:18.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: overall HEALTH_OK 2023-07-14T18:05:18.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:18.634 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-14T18:05:18.635 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph config generate-minimal-conf 2023-07-14T18:05:19.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:18 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:19.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:18 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/1318728866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:05:19.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:18 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:19.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:18 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:19.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:18 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:19.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:18 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/1318728866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:05:19.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:18 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:19.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:18 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:19.198 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:05:19.362 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:18 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:19.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:18 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/1318728866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:05:19.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:18 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:05:19.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:18 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:19 smithi115 ceph-mon[127451]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:20.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:20.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:20.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:19 smithi187 ceph-mon[129317]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:20.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.175 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:20.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:20.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:19 smithi150 ceph-mon[128853]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:20.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:20.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:20.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:20.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.424+0000 7f7edfba3700 1 -- 172.21.15.115:0/1991288939 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed80ff690 msgr2=0x7f7ed80ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:20.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.424+0000 7f7edfba3700 1 --2- 172.21.15.115:0/1991288939 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed80ff690 0x7f7ed80ffa60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009910 tx=0x7f7ed4009c20 comp rx=0 tx=0).stop 2023-07-14T18:05:20.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.424+0000 7f7edfba3700 1 -- 172.21.15.115:0/1991288939 shutdown_connections 2023-07-14T18:05:20.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.424+0000 7f7edfba3700 1 --2- 172.21.15.115:0/1991288939 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed80ff690 0x7f7ed80ffa60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:20.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.424+0000 7f7edfba3700 1 -- 172.21.15.115:0/1991288939 >> 172.21.15.115:0/1991288939 conn(0x7f7ed8075070 msgr2=0x7f7ed8075470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:20.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 -- 172.21.15.115:0/1991288939 shutdown_connections 2023-07-14T18:05:20.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 -- 172.21.15.115:0/1991288939 wait complete. 2023-07-14T18:05:20.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 Processor -- start 2023-07-14T18:05:20.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 -- start start 2023-07-14T18:05:20.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed80ff690 0x7f7ed8195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:20.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed8196140 0x7f7ed81965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:20.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ed819ec70 0x7f7ed81a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:20.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed81a17d0 con 0x7f7ed8196140 2023-07-14T18:05:20.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed81a1940 con 0x7f7ed80ff690 2023-07-14T18:05:20.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.425+0000 7f7edfba3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed81a1b00 con 0x7f7ed819ec70 2023-07-14T18:05:20.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd13e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed8196140 0x7f7ed81965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:20.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd13e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed8196140 0x7f7ed81965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51644/0 (socket says 172.21.15.115:51644) 2023-07-14T18:05:20.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd13e700 1 -- 172.21.15.115:0/2152682790 learned_addr learned my addr 172.21.15.115:0/2152682790 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:05:20.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd93f700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed80ff690 0x7f7ed8195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:20.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7ede140700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ed819ec70 0x7f7ed81a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:20.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd13e700 1 -- 172.21.15.115:0/2152682790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ed819ec70 msgr2=0x7f7ed81a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:20.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd13e700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ed819ec70 0x7f7ed81a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:20.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd13e700 1 -- 172.21.15.115:0/2152682790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed80ff690 msgr2=0x7f7ed8195c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:20.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd13e700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed80ff690 0x7f7ed8195c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:20.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd13e700 1 -- 172.21.15.115:0/2152682790 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed40095f0 con 0x7f7ed8196140 2023-07-14T18:05:20.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edd13e700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed8196140 0x7f7ed81965b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800b780 tx=0x7f7ec800ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:20.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7eceffd700 1 -- 172.21.15.115:0/2152682790 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8004230 con 0x7f7ed8196140 2023-07-14T18:05:20.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7eceffd700 1 -- 172.21.15.115:0/2152682790 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7ec8004390 con 0x7f7ed8196140 2023-07-14T18:05:20.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed81a1de0 con 0x7f7ed8196140 2023-07-14T18:05:20.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7eceffd700 1 -- 172.21.15.115:0/2152682790 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8011a30 con 0x7f7ed8196140 2023-07-14T18:05:20.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.426+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed81a22b0 con 0x7f7ed8196140 2023-07-14T18:05:20.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.427+0000 7f7eceffd700 1 -- 172.21.15.115:0/2152682790 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f7ec8011c90 con 0x7f7ed8196140 2023-07-14T18:05:20.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.427+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed8061c70 con 0x7f7ed8196140 2023-07-14T18:05:20.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.427+0000 7f7eceffd700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7ec4038c80 0x7f7ec403b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:20.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.427+0000 7f7eceffd700 1 -- 172.21.15.115:0/2152682790 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ec804ae00 con 0x7f7ed8196140 2023-07-14T18:05:20.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.427+0000 7f7edd93f700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7ec4038c80 0x7f7ec403b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:20.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.428+0000 7f7edd93f700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7ec4038c80 0x7f7ec403b130 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400b5c0 tx=0x7f7ed40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:20.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.432+0000 7f7eceffd700 1 -- 172.21.15.115:0/2152682790 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7ec8018ad0 con 0x7f7ed8196140 2023-07-14T18:05:20.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.630+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f7ed81a26d0 con 0x7f7ed8196140 2023-07-14T18:05:20.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.630+0000 7f7eceffd700 1 -- 172.21.15.115:0/2152682790 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f7ec80186e0 con 0x7f7ed8196140 2023-07-14T18:05:20.661 INFO:teuthology.orchestra.run.smithi115.stdout:# minimal ceph.conf for a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:05:20.661 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2023-07-14T18:05:20.661 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:05:20.661 INFO:teuthology.orchestra.run.smithi115.stdout: mon_host = [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] 2023-07-14T18:05:20.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7ec4038c80 msgr2=0x7f7ec403b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:20.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7ec4038c80 0x7f7ec403b130 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400b5c0 tx=0x7f7ed40058e0 comp rx=0 tx=0).stop 2023-07-14T18:05:20.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed8196140 msgr2=0x7f7ed81965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:20.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed8196140 0x7f7ed81965b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800b780 tx=0x7f7ec800ba90 comp rx=0 tx=0).stop 2023-07-14T18:05:20.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 shutdown_connections 2023-07-14T18:05:20.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7ec4038c80 0x7f7ec403b130 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:20.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ed819ec70 0x7f7ed81a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:20.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ed80ff690 0x7f7ed8195c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:20.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 --2- 172.21.15.115:0/2152682790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ed8196140 0x7f7ed81965b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:20.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 >> 172.21.15.115:0/2152682790 conn(0x7f7ed8075070 msgr2=0x7f7ed80fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:20.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 shutdown_connections 2023-07-14T18:05:20.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:20.632+0000 7f7edfba3700 1 -- 172.21.15.115:0/2152682790 wait complete. 2023-07-14T18:05:21.044 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-14T18:05:21.044 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:05:21.045 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T18:05:21.076 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:05:21.076 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:05:21.143 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T18:05:21.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T18:05:21.179 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T18:05:21.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:05:21.247 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: Updating smithi115:/etc/ceph/ceph.conf 2023-07-14T18:05:21.247 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: Updating smithi150:/etc/ceph/ceph.conf 2023-07-14T18:05:21.247 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: Updating smithi187:/etc/ceph/ceph.conf 2023-07-14T18:05:21.247 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: Updating smithi150:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:21.247 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: Updating smithi115:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:21.247 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: Updating smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:21.248 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.248 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.248 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.248 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.248 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.248 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.248 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.249 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T18:05:21.249 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:21.249 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:05:21.249 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:21.249 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: Reconfiguring daemon mon.a on smithi115 2023-07-14T18:05:21.249 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:20 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2152682790' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:21.250 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T18:05:21.250 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T18:05:21.251 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: Updating smithi115:/etc/ceph/ceph.conf 2023-07-14T18:05:21.251 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: Updating smithi150:/etc/ceph/ceph.conf 2023-07-14T18:05:21.251 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: Updating smithi187:/etc/ceph/ceph.conf 2023-07-14T18:05:21.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: Updating smithi150:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:21.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: Updating smithi115:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:21.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: Updating smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:21.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T18:05:21.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:21.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:05:21.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:21.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: Reconfiguring daemon mon.a on smithi115 2023-07-14T18:05:21.254 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:20 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2152682790' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:21.283 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: Updating smithi115:/etc/ceph/ceph.conf 2023-07-14T18:05:21.283 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: Updating smithi150:/etc/ceph/ceph.conf 2023-07-14T18:05:21.283 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: Updating smithi187:/etc/ceph/ceph.conf 2023-07-14T18:05:21.283 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: Updating smithi150:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:21.283 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: Updating smithi115:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:21.283 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: Updating smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:05:21.284 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.284 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.284 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.284 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.284 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.284 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.284 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:21.285 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T18:05:21.285 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:21.285 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:05:21.285 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:21.285 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: Reconfiguring daemon mon.a on smithi115 2023-07-14T18:05:21.285 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:20 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2152682790' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:21.287 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T18:05:21.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:05:21.356 INFO:tasks.cephadm:Adding mgr.x on smithi115 2023-07-14T18:05:21.357 INFO:tasks.cephadm:Adding mgr.y on smithi150 2023-07-14T18:05:21.357 INFO:tasks.cephadm:Adding mgr.z on smithi187 2023-07-14T18:05:21.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch apply mgr '3;smithi115=x;smithi150=y;smithi187=z' 2023-07-14T18:05:21.947 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:05:22.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:21 smithi115 ceph-mon[127451]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:22.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:21 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:22.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:21 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:22.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:21 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:22.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:21 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:05:22.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:21 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:22.349 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:21 smithi187 ceph-mon[129317]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:22.349 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:21 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:22.349 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:21 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:22.349 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:21 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:22.349 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:21 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:05:22.349 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:21 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:22.350 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:21 smithi150 ceph-mon[128853]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:22.350 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:21 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:22.350 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:21 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:22.351 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:21 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:22.351 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:21 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:05:22.351 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:21 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:23.140 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.109+0000 7f7091417700 1 -- 172.21.15.187:0/910947293 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f708c101a30 msgr2=0x7f708c101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:23.140 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.109+0000 7f7091417700 1 --2- 172.21.15.187:0/910947293 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f708c101a30 0x7f708c101e40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f707c009910 tx=0x7f707c009c20 comp rx=0 tx=0).stop 2023-07-14T18:05:23.140 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.109+0000 7f7091417700 1 -- 172.21.15.187:0/910947293 shutdown_connections 2023-07-14T18:05:23.140 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.109+0000 7f7091417700 1 --2- 172.21.15.187:0/910947293 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f708c101a30 0x7f708c101e40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:23.141 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.109+0000 7f7091417700 1 -- 172.21.15.187:0/910947293 >> 172.21.15.187:0/910947293 conn(0x7f708c0fcff0 msgr2=0x7f708c0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:23.141 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.109+0000 7f7091417700 1 -- 172.21.15.187:0/910947293 shutdown_connections 2023-07-14T18:05:23.141 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.109+0000 7f7091417700 1 -- 172.21.15.187:0/910947293 wait complete. 2023-07-14T18:05:23.141 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.109+0000 7f7091417700 1 Processor -- start 2023-07-14T18:05:23.144 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f7091417700 1 -- start start 2023-07-14T18:05:23.145 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f7091417700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f708c101a30 0x7f708c196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:23.145 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f7091417700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f708c1971f0 0x7f708c19c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:23.145 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f7091417700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708c197660 0x7f708c19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:23.145 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f7091417700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c19efb0 con 0x7f708c101a30 2023-07-14T18:05:23.146 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f7091417700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c19f120 con 0x7f708c197660 2023-07-14T18:05:23.146 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f7091417700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c19f2e0 con 0x7f708c1971f0 2023-07-14T18:05:23.146 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f708c1971f0 0x7f708c19c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:23.147 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708a7fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f708c1971f0 0x7f708c19c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:58446/0 (socket says 172.21.15.187:58446) 2023-07-14T18:05:23.147 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708a7fc700 1 -- 172.21.15.187:0/2936487403 learned_addr learned my addr 172.21.15.187:0/2936487403 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:05:23.147 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708affd700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f708c101a30 0x7f708c196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:23.147 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708b7fe700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708c197660 0x7f708c19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:23.147 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708a7fc700 1 -- 172.21.15.187:0/2936487403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708c197660 msgr2=0x7f708c19e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:23.148 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708a7fc700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708c197660 0x7f708c19e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:23.148 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708a7fc700 1 -- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f708c101a30 msgr2=0x7f708c196cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:23.148 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708a7fc700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f708c101a30 0x7f708c196cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:23.148 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708a7fc700 1 -- 172.21.15.187:0/2936487403 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c0095f0 con 0x7f708c1971f0 2023-07-14T18:05:23.148 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.110+0000 7f708a7fc700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f708c1971f0 0x7f708c19c250 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f708000b780 tx=0x7f708000ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:23.148 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.111+0000 7f7073fff700 1 -- 172.21.15.187:0/2936487403 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080004230 con 0x7f708c1971f0 2023-07-14T18:05:23.148 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.111+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708c19f5c0 con 0x7f708c1971f0 2023-07-14T18:05:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.111+0000 7f7073fff700 1 -- 172.21.15.187:0/2936487403 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7080004390 con 0x7f708c1971f0 2023-07-14T18:05:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.111+0000 7f7073fff700 1 -- 172.21.15.187:0/2936487403 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7080011a50 con 0x7f708c1971f0 2023-07-14T18:05:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.111+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708c19fb90 con 0x7f708c1971f0 2023-07-14T18:05:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.111+0000 7f7073fff700 1 -- 172.21.15.187:0/2936487403 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f7080011bb0 con 0x7f708c1971f0 2023-07-14T18:05:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.111+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708c190460 con 0x7f708c1971f0 2023-07-14T18:05:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.112+0000 7f7073fff700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7074038c30 0x7f707403b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.112+0000 7f7073fff700 1 -- 172.21.15.187:0/2936487403 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f708004aec0 con 0x7f708c1971f0 2023-07-14T18:05:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.112+0000 7f708affd700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7074038c30 0x7f707403b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:23.150 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.112+0000 7f708affd700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7074038c30 0x7f707403b0e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f707c009fd0 tx=0x7f707c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:23.150 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.117+0000 7f7073fff700 1 -- 172.21.15.187:0/2936487403 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f708001ae40 con 0x7f708c1971f0 2023-07-14T18:05:23.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:22 smithi187 ceph-mon[129317]: Reconfiguring mon.b (monmap changed)... 2023-07-14T18:05:23.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:22 smithi187 ceph-mon[129317]: Reconfiguring daemon mon.b on smithi150 2023-07-14T18:05:23.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:23.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:23.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:23.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:05:23.274 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:23.347 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:22 smithi115 ceph-mon[127451]: Reconfiguring mon.b (monmap changed)... 2023-07-14T18:05:23.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:22 smithi115 ceph-mon[127451]: Reconfiguring daemon mon.b on smithi150 2023-07-14T18:05:23.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:23.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:23.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:23.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:05:23.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:23.356 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.325+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi115=x;smithi150=y;smithi187=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f708c060b10 con 0x7f7074038c30 2023-07-14T18:05:23.362 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:22 smithi150 ceph-mon[128853]: Reconfiguring mon.b (monmap changed)... 2023-07-14T18:05:23.362 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:22 smithi150 ceph-mon[128853]: Reconfiguring daemon mon.b on smithi150 2023-07-14T18:05:23.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:23.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:23.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:05:23.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:05:23.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:23.364 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.332+0000 7f7073fff700 1 -- 172.21.15.187:0/2936487403 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f708c060b10 con 0x7f7074038c30 2023-07-14T18:05:23.364 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mgr update... 2023-07-14T18:05:23.364 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7074038c30 msgr2=0x7f707403b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:23.365 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7074038c30 0x7f707403b0e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f707c009fd0 tx=0x7f707c005fd0 comp rx=0 tx=0).stop 2023-07-14T18:05:23.365 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f708c1971f0 msgr2=0x7f708c19c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:23.365 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f708c1971f0 0x7f708c19c250 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f708000b780 tx=0x7f708000ba90 comp rx=0 tx=0).stop 2023-07-14T18:05:23.365 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 shutdown_connections 2023-07-14T18:05:23.365 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7074038c30 0x7f707403b0e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:23.365 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708c197660 0x7f708c19e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:23.366 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f708c101a30 0x7f708c196cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:23.366 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 --2- 172.21.15.187:0/2936487403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f708c1971f0 0x7f708c19c250 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:23.366 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 >> 172.21.15.187:0/2936487403 conn(0x7f708c0fcff0 msgr2=0x7f708c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:23.366 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 shutdown_connections 2023-07-14T18:05:23.366 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:05:23.334+0000 7f7091417700 1 -- 172.21.15.187:0/2936487403 wait complete. 2023-07-14T18:05:23.803 DEBUG:teuthology.orchestra.run.smithi150:mgr.y> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.y.service 2023-07-14T18:05:23.806 DEBUG:teuthology.orchestra.run.smithi187:mgr.z> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.z.service 2023-07-14T18:05:23.808 INFO:tasks.cephadm:Deploying OSDs... 2023-07-14T18:05:23.808 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:05:23.808 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T18:05:23.829 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T18:05:23.829 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2023-07-14T18:05:23.836 INFO:journalctl@ceph.mgr.z.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:05:23.839 INFO:journalctl@ceph.mgr.y.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:05:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T18:05:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:23.889 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 380263 Links: 1 2023-07-14T18:05:23.889 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:23.889 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:23.889 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-07-14 18:04:38.612684430 +0000 2023-07-14T18:05:23.889 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-07-14 17:53:18.598881103 +0000 2023-07-14T18:05:23.890 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-07-14 17:53:18.598881103 +0000 2023-07-14T18:05:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-07-14T18:05:23.890 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T18:05:23.958 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-07-14T18:05:23.958 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-07-14T18:05:23.958 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000254618 s, 2.0 MB/s 2023-07-14T18:05:23.959 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T18:05:24.019 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2023-07-14T18:05:24.078 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T18:05:24.078 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:24.078 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 380321 Links: 1 2023-07-14T18:05:24.078 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:24.078 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:24.078 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-07-14 18:04:38.612684430 +0000 2023-07-14T18:05:24.079 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-07-14 17:53:19.135871992 +0000 2023-07-14T18:05:24.079 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-07-14 17:53:19.135871992 +0000 2023-07-14T18:05:24.079 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-07-14T18:05:24.079 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T18:05:24.112 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:23 smithi150 ceph-mon[128853]: Reconfiguring mon.c (monmap changed)... 2023-07-14T18:05:24.113 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:23 smithi150 ceph-mon[128853]: Reconfiguring daemon mon.c on smithi187 2023-07-14T18:05:24.113 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:23 smithi150 ceph-mon[128853]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:24.113 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:23 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:24.145 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-07-14T18:05:24.146 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-07-14T18:05:24.146 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000234549 s, 2.2 MB/s 2023-07-14T18:05:24.147 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T18:05:24.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:23 smithi187 ceph-mon[129317]: Reconfiguring mon.c (monmap changed)... 2023-07-14T18:05:24.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:23 smithi187 ceph-mon[129317]: Reconfiguring daemon mon.c on smithi187 2023-07-14T18:05:24.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:23 smithi187 ceph-mon[129317]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:24.163 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:23 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:24.205 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2023-07-14T18:05:24.263 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T18:05:24.263 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:24.263 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 383281 Links: 1 2023-07-14T18:05:24.263 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:24.263 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:24.263 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-07-14 18:04:38.612684430 +0000 2023-07-14T18:05:24.264 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-07-14 17:53:19.638863457 +0000 2023-07-14T18:05:24.264 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-07-14 17:53:19.638863457 +0000 2023-07-14T18:05:24.264 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-07-14T18:05:24.264 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T18:05:24.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:23 smithi115 ceph-mon[127451]: Reconfiguring mon.c (monmap changed)... 2023-07-14T18:05:24.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:23 smithi115 ceph-mon[127451]: Reconfiguring daemon mon.c on smithi187 2023-07-14T18:05:24.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:23 smithi115 ceph-mon[127451]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:24.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:23 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:24.330 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-07-14T18:05:24.330 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-07-14T18:05:24.330 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000246166 s, 2.1 MB/s 2023-07-14T18:05:24.331 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T18:05:24.388 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2023-07-14T18:05:24.445 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T18:05:24.445 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:24.445 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 384161 Links: 1 2023-07-14T18:05:24.445 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:24.446 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:24.446 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-07-14 18:04:38.612684430 +0000 2023-07-14T18:05:24.446 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-07-14 17:53:20.171854413 +0000 2023-07-14T18:05:24.446 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-07-14 17:53:20.171854413 +0000 2023-07-14T18:05:24.446 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-07-14T18:05:24.446 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T18:05:24.510 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-07-14T18:05:24.510 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-07-14T18:05:24.510 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000194312 s, 2.6 MB/s 2023-07-14T18:05:24.511 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T18:05:24.568 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-07-14T18:05:24.569 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T18:05:24.589 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T18:05:24.589 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2023-07-14T18:05:24.649 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T18:05:24.649 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:24.649 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 382069 Links: 1 2023-07-14T18:05:24.650 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:24.650 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:24.650 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-07-14 18:05:12.220362513 +0000 2023-07-14T18:05:24.650 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-07-14 17:57:48.597884005 +0000 2023-07-14T18:05:24.650 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-07-14 17:57:48.597884005 +0000 2023-07-14T18:05:24.650 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-07-14T18:05:24.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T18:05:24.720 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-07-14T18:05:24.720 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-07-14T18:05:24.720 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000265013 s, 1.9 MB/s 2023-07-14T18:05:24.721 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T18:05:24.782 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2023-07-14T18:05:24.842 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T18:05:24.842 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:24.843 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 379304 Links: 1 2023-07-14T18:05:24.843 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:24.843 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:24.843 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-07-14 18:05:12.220362513 +0000 2023-07-14T18:05:24.843 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-07-14 17:57:49.133873863 +0000 2023-07-14T18:05:24.843 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-07-14 17:57:49.133873863 +0000 2023-07-14T18:05:24.843 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-07-14T18:05:24.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T18:05:24.911 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-07-14T18:05:24.912 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-07-14T18:05:24.912 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000240318 s, 2.1 MB/s 2023-07-14T18:05:24.913 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T18:05:24.972 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2023-07-14T18:05:25.031 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T18:05:25.031 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:25.031 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 375739 Links: 1 2023-07-14T18:05:25.031 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:25.032 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:25.032 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-07-14 18:05:12.220362513 +0000 2023-07-14T18:05:25.032 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-07-14 17:57:49.647864137 +0000 2023-07-14T18:05:25.032 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-07-14 17:57:49.647864137 +0000 2023-07-14T18:05:25.032 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-07-14T18:05:25.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T18:05:25.099 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-07-14T18:05:25.099 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-07-14T18:05:25.099 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000236131 s, 2.2 MB/s 2023-07-14T18:05:25.100 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T18:05:25.160 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2023-07-14T18:05:25.219 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T18:05:25.219 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:25.220 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 379422 Links: 1 2023-07-14T18:05:25.220 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:25.220 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:25.220 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-07-14 18:05:12.220362513 +0000 2023-07-14T18:05:25.220 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-07-14 17:57:50.163854373 +0000 2023-07-14T18:05:25.220 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-07-14 17:57:50.163854373 +0000 2023-07-14T18:05:25.220 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-07-14T18:05:25.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T18:05:25.287 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-07-14T18:05:25.288 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-07-14T18:05:25.288 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000232362 s, 2.2 MB/s 2023-07-14T18:05:25.289 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T18:05:25.349 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-07-14T18:05:25.350 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T18:05:25.362 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi115=x;smithi150=y;smithi187=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:25.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: Saving service mgr spec with placement smithi115=x;smithi150=y;smithi187=z;count:3 2023-07-14T18:05:25.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:25.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:25.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:25.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:25.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:25.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:25.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:05:25.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:05:25.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:05:25.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:25.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:25 smithi150 ceph-mon[128853]: Deploying daemon mgr.z on smithi187 2023-07-14T18:05:25.366 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T18:05:25.367 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2023-07-14T18:05:25.419 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi115=x;smithi150=y;smithi187=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:25.419 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: Saving service mgr spec with placement smithi115=x;smithi150=y;smithi187=z;count:3 2023-07-14T18:05:25.419 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:25.419 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:25.419 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:25.419 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:25.420 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:25.420 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:25.420 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:05:25.420 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:05:25.420 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:05:25.420 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:25.420 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:25 smithi187 ceph-mon[129317]: Deploying daemon mgr.z on smithi187 2023-07-14T18:05:25.424 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T18:05:25.424 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:25.425 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 381061 Links: 1 2023-07-14T18:05:25.425 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:25.425 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:25.425 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-07-14 18:05:04.024112557 +0000 2023-07-14T18:05:25.425 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-07-14 17:53:24.864883752 +0000 2023-07-14T18:05:25.425 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-07-14 17:53:24.864883752 +0000 2023-07-14T18:05:25.425 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2023-07-14T18:05:25.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T18:05:25.490 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-07-14T18:05:25.491 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-07-14T18:05:25.491 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000242085 s, 2.1 MB/s 2023-07-14T18:05:25.492 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T18:05:25.551 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2023-07-14T18:05:25.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi115=x;smithi150=y;smithi187=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:25.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: Saving service mgr spec with placement smithi115=x;smithi150=y;smithi187=z;count:3 2023-07-14T18:05:25.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:25.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:25.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:25.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:25.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:25.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:25.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:05:25.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:05:25.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:05:25.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:25.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:25 smithi115 ceph-mon[127451]: Deploying daemon mgr.z on smithi187 2023-07-14T18:05:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T18:05:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:25.608 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 368463 Links: 1 2023-07-14T18:05:25.608 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:25.609 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:25.609 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-07-14 18:05:25.530185144 +0000 2023-07-14T18:05:25.609 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-07-14 17:53:25.391886310 +0000 2023-07-14T18:05:25.609 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-07-14 17:53:25.391886310 +0000 2023-07-14T18:05:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2023-07-14T18:05:25.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T18:05:25.742 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-07-14T18:05:25.742 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-07-14T18:05:25.742 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000244612 s, 2.1 MB/s 2023-07-14T18:05:25.743 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T18:05:25.761 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2023-07-14T18:05:25.820 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T18:05:25.820 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:25.820 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 377634 Links: 1 2023-07-14T18:05:25.820 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:25.820 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:25.821 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-07-14 18:05:25.665185595 +0000 2023-07-14T18:05:25.821 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-07-14 17:53:25.933888941 +0000 2023-07-14T18:05:25.821 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-07-14 17:53:25.933888941 +0000 2023-07-14T18:05:25.821 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2023-07-14T18:05:25.821 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T18:05:25.890 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-07-14T18:05:25.890 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-07-14T18:05:25.890 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000210939 s, 2.4 MB/s 2023-07-14T18:05:25.891 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T18:05:25.948 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2023-07-14T18:05:26.004 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T18:05:26.005 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:05:26.005 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 6h/6d Inode: 382197 Links: 1 2023-07-14T18:05:26.005 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:05:26.005 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:05:26.005 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-07-14 18:05:25.665185595 +0000 2023-07-14T18:05:26.005 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-07-14 17:53:26.451891455 +0000 2023-07-14T18:05:26.005 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-07-14 17:53:26.451891455 +0000 2023-07-14T18:05:26.006 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: - 2023-07-14T18:05:26.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T18:05:26.032 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:25 smithi187 systemd[1]: Starting Ceph mgr.z for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:05:26.058 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-07-14T18:05:26.058 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-07-14T18:05:26.058 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.00021064 s, 2.4 MB/s 2023-07-14T18:05:26.059 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T18:05:26.117 INFO:tasks.cephadm:Deploying osd.0 on smithi115 with /dev/vg_nvme/lv_4... 2023-07-14T18:05:26.118 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T18:05:26.283 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:26 smithi187 ceph-mon[129317]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:26.283 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:26 smithi187 podman[131362]: 2023-07-14T18:05:26.572 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:26 smithi187 bash[131362]: 268066ecc647267dab1fcf6853df9ad02e0278f64480141d0059299f2d871390 2023-07-14T18:05:26.572 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:26 smithi187 ceph-mgr[131486]: -- 172.21.15.187:0/3388865100 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563db639b1e0 con 0x563db6370800 2023-07-14T18:05:26.572 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:26 smithi187 systemd[1]: Started Ceph mgr.z for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:05:26.572 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:26.375+0000 7f12ec5b8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:05:26.573 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:26.454+0000 7f12ec5b8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:05:26.573 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:26.541+0000 7f12ec5b8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:05:26.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:26 smithi115 ceph-mon[127451]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:26.612 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:26 smithi150 ceph-mon[128853]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:26.679 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:05:26.831 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:26.633+0000 7f12ec5b8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:05:27.135 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:26.801+0000 7f12ec5b8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:05:27.391 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:27.104+0000 7f12ec5b8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:05:27.391 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:27.192+0000 7f12ec5b8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:05:27.392 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:27.320+0000 7f12ec5b8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:05:27.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:27.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:27 smithi115 ceph-mon[127451]: Deploying daemon mgr.y on smithi150 2023-07-14T18:05:27.599 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:27 smithi115 ceph-mon[127451]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:27.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:27.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:27.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:27.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:05:27.613 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:05:27.614 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:05:27.614 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:27.614 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:27 smithi150 ceph-mon[128853]: Deploying daemon mgr.y on smithi150 2023-07-14T18:05:27.614 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:27 smithi150 ceph-mon[128853]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:27.675 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:27.415+0000 7f12ec5b8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:05:27.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:27.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:27.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:27.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:05:27.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:05:27.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:05:27.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:27.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-mon[129317]: Deploying daemon mgr.y on smithi150 2023-07-14T18:05:27.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-mon[129317]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:28.065 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:27 smithi150 systemd[1]: Starting Ceph mgr.y for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:05:28.175 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:27.831+0000 7f12ec5b8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:05:28.200 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T18:05:28.200 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T18:05:28.200 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2023-07-14T18:05:28.200 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2023-07-14T18:05:28.200 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129137 s, 812 MB/s 2023-07-14T18:05:28.200 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2023-07-14T18:05:28.353 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:28 smithi150 podman[130292]: 2023-07-14T18:05:28.353 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:28 smithi150 bash[130292]: 28b0dc3744a81e88a838a5e60ebcd13d063e326c5d0201bfc853cf47268da547 2023-07-14T18:05:28.353 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:28 smithi150 ceph-mgr[130333]: -- 172.21.15.150:0/124573121 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56061dcc31e0 con 0x56061dc98c00 2023-07-14T18:05:28.613 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:28 smithi150 systemd[1]: Started Ceph mgr.y for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:05:28.613 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:28.389+0000 7f756f19b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:05:28.613 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:28.472+0000 7f756f19b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:05:28.613 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:28.564+0000 7f756f19b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:05:28.622 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_4 2023-07-14T18:05:28.925 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:28.400+0000 7f12ec5b8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:05:28.925 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:28.581+0000 7f12ec5b8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:05:29.113 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:28.663+0000 7f756f19b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:05:29.113 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:28.846+0000 7f756f19b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:05:29.173 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:05:29.533 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:29.175+0000 7f756f19b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:05:29.533 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:29.265+0000 7f756f19b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:05:29.533 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:29.401+0000 7f756f19b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:05:29.533 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.533 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.534 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.534 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.534 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:29.534 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-mon[128853]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:29.534 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.597 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:29.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:29 smithi115 ceph-mon[127451]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:29.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.675 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:29.394+0000 7f12ec5b8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:05:29.676 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:29.490+0000 7f12ec5b8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:05:29.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:29.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-mon[129317]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:29.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:29.863 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:29.502+0000 7f756f19b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:05:29.993 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:29.671+0000 7f12ec5b8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:05:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.222+0000 7f7608e0b700 1 -- 172.21.15.115:0/1398690208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fe530 msgr2=0x7f76040fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:30.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.222+0000 7f7608e0b700 1 --2- 172.21.15.115:0/1398690208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fe530 0x7f76040fe940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009990 tx=0x7f75f4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:05:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.222+0000 7f7608e0b700 1 -- 172.21.15.115:0/1398690208 shutdown_connections 2023-07-14T18:05:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.222+0000 7f7608e0b700 1 --2- 172.21.15.115:0/1398690208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76040ff830 0x7f76040ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.222+0000 7f7608e0b700 1 --2- 172.21.15.115:0/1398690208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fe530 0x7f76040fe940 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.222+0000 7f7608e0b700 1 --2- 172.21.15.115:0/1398690208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76040fee80 0x7f76040ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.222+0000 7f7608e0b700 1 -- 172.21.15.115:0/1398690208 >> 172.21.15.115:0/1398690208 conn(0x7f76040fa1e0 msgr2=0x7f76040fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:30.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.222+0000 7f7608e0b700 1 -- 172.21.15.115:0/1398690208 shutdown_connections 2023-07-14T18:05:30.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.222+0000 7f7608e0b700 1 -- 172.21.15.115:0/1398690208 wait complete. 2023-07-14T18:05:30.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7608e0b700 1 Processor -- start 2023-07-14T18:05:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7608e0b700 1 -- start start 2023-07-14T18:05:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7608e0b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76040fe530 0x7f7604197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7608e0b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fee80 0x7f7604198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7608e0b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76040ff830 0x7f760419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7608e0b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760419f670 con 0x7f76040fee80 2023-07-14T18:05:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7608e0b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760419f7b0 con 0x7f76040ff830 2023-07-14T18:05:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7608e0b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760419f940 con 0x7f76040fe530 2023-07-14T18:05:30.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7601d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fee80 0x7f7604198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7601d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fee80 0x7f7604198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51656/0 (socket says 172.21.15.115:51656) 2023-07-14T18:05:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7601d9b700 1 -- 172.21.15.115:0/4150652548 learned_addr learned my addr 172.21.15.115:0/4150652548 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:05:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f760259c700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76040fe530 0x7f7604197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.223+0000 7f7602d9d700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76040ff830 0x7f760419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f7601d9b700 1 -- 172.21.15.115:0/4150652548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76040fe530 msgr2=0x7f7604197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f7601d9b700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76040fe530 0x7f7604197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:30.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f7601d9b700 1 -- 172.21.15.115:0/4150652548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76040ff830 msgr2=0x7f760419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f7601d9b700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76040ff830 0x7f760419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f7601d9b700 1 -- 172.21.15.115:0/4150652548 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40095f0 con 0x7f76040fee80 2023-07-14T18:05:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f7602d9d700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76040ff830 0x7f760419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:05:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f7601d9b700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fee80 0x7f7604198020 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f75f800e9f0 tx=0x7f75f800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f75f37fe700 1 -- 172.21.15.115:0/4150652548 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f80097b0 con 0x7f76040fee80 2023-07-14T18:05:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f75f37fe700 1 -- 172.21.15.115:0/4150652548 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f75f8004cc0 con 0x7f76040fee80 2023-07-14T18:05:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760419fc20 con 0x7f76040fee80 2023-07-14T18:05:30.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f75f37fe700 1 -- 172.21.15.115:0/4150652548 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8005560 con 0x7f76040fee80 2023-07-14T18:05:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.224+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76041a00d0 con 0x7f76040fee80 2023-07-14T18:05:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.225+0000 7f75f37fe700 1 -- 172.21.15.115:0/4150652548 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f75f8021440 con 0x7f76040fee80 2023-07-14T18:05:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.225+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7604109310 con 0x7f76040fee80 2023-07-14T18:05:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.225+0000 7f75f37fe700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75ec0389e0 0x7f75ec03ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.225+0000 7f75f37fe700 1 -- 172.21.15.115:0/4150652548 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f75f8014070 con 0x7f76040fee80 2023-07-14T18:05:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.225+0000 7f760259c700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75ec0389e0 0x7f75ec03ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:30.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.225+0000 7f760259c700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75ec0389e0 0x7f75ec03ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f75f4006010 tx=0x7f75f4005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:30.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.230+0000 7f75f37fe700 1 -- 172.21.15.115:0/4150652548 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f75f8005700 con 0x7f76040fee80 2023-07-14T18:05:30.363 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:29.959+0000 7f756f19b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:05:30.374 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:29.962+0000 7f12ec5b8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:05:30.375 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:30.197+0000 7f12ec5b8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:05:30.375 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:30.344+0000 7f12ec5b8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:05:30.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:30.435+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7604060a80 con 0x7f75ec0389e0 2023-07-14T18:05:30.863 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:30.544+0000 7f756f19b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:05:30.863 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:30.714+0000 7f756f19b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:05:30.925 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:30.601+0000 7f12ec5b8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:05:31.065 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.065 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.065 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:31.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:31.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T18:05:31.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:05:31.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:05:31.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:31.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: Reconfiguring daemon mgr.x on smithi115 2023-07-14T18:05:31.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:05:31.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:05:31.067 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:31.331 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.332 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.332 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.332 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.332 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:31.332 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:31.332 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.333 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T18:05:31.333 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:05:31.333 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:05:31.333 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:31.333 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: Reconfiguring daemon mgr.x on smithi115 2023-07-14T18:05:31.333 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:05:31.333 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:05:31.334 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:31.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:31.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:31.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:31.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T18:05:31.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:05:31.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:05:31.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:31.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: Reconfiguring daemon mgr.x on smithi115 2023-07-14T18:05:31.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:05:31.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:05:31.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:31.649 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:31.300+0000 7f12ec5b8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:05:31.649 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:31.429+0000 7f12ec5b8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:05:31.649 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:31.618+0000 7f12ec5b8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:05:31.769 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:31 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:31.478+0000 7f756f19b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:05:31.769 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:31 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:31.569+0000 7f756f19b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:05:31.925 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:31.730+0000 7f12ec5b8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:05:32.049 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:31 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:31.737+0000 7f756f19b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:05:32.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:32 smithi115 ceph-mon[127451]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:32.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:32 smithi115 ceph-mon[127451]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:32.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:32.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:32.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:32.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:32.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:32.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:32.363 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:32.017+0000 7f756f19b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:05:32.363 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:32.246+0000 7f756f19b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:05:32.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-mon[128853]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:32.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-mon[128853]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:32.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:32.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:32.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:32.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:32.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:32.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:32.425 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:31.943+0000 7f12ec5b8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:05:32.425 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:32 smithi187 ceph-mon[129317]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:32.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:32 smithi187 ceph-mon[129317]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:32.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:32.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:32.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:32.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:32.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:32.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:32.683 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:32.393+0000 7f756f19b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:05:33.113 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:32 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:32.651+0000 7f756f19b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:05:33.175 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:05:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z[131440]: 2023-07-14T18:05:32.797+0000 7f12ec5b8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:05:33.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:33.348+0000 7f75f37fe700 1 -- 172.21.15.115:0/4150652548 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 14) v1 ==== 90600+0+0 (secure 0 0 0) 0x7f75f801dd20 con 0x7f76040fee80 2023-07-14T18:05:33.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: Standby manager daemon z started 2023-07-14T18:05:33.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T18:05:33.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:05:33.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T18:05:33.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:05:33.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:33.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2915897618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "578188a6-96fa-444a-8a11-224bb5a7c098"}]: dispatch 2023-07-14T18:05:33.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2915897618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "578188a6-96fa-444a-8a11-224bb5a7c098"}]': finished 2023-07-14T18:05:33.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T18:05:33.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:33.707 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:33.356+0000 7f756f19b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:05:33.707 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:33.486+0000 7f756f19b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:05:33.707 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: Standby manager daemon z started 2023-07-14T18:05:33.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T18:05:33.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:05:33.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T18:05:33.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:05:33.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:33.708 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2915897618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "578188a6-96fa-444a-8a11-224bb5a7c098"}]: dispatch 2023-07-14T18:05:33.709 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2915897618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "578188a6-96fa-444a-8a11-224bb5a7c098"}]': finished 2023-07-14T18:05:33.709 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T18:05:33.709 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:33.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: Standby manager daemon z started 2023-07-14T18:05:33.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T18:05:33.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:05:33.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T18:05:33.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: from='mgr.? 172.21.15.187:0/1441645171' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:05:33.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:33.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2915897618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "578188a6-96fa-444a-8a11-224bb5a7c098"}]: dispatch 2023-07-14T18:05:33.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2915897618' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "578188a6-96fa-444a-8a11-224bb5a7c098"}]': finished 2023-07-14T18:05:33.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T18:05:33.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:34.034 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:33.675+0000 7f756f19b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:05:34.034 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:33.789+0000 7f756f19b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:05:34.363 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:34 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:34.003+0000 7f756f19b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:05:34.675 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:34 smithi187 ceph-mon[129317]: mgrmap e14: x(active, since 64s), standbys: z 2023-07-14T18:05:34.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:34 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T18:05:34.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:34 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1816695321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:05:34.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:34 smithi115 ceph-mon[127451]: mgrmap e14: x(active, since 64s), standbys: z 2023-07-14T18:05:34.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:34 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T18:05:34.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:34 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1816695321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:05:34.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:34 smithi150 ceph-mon[128853]: mgrmap e14: x(active, since 64s), standbys: z 2023-07-14T18:05:34.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:34 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T18:05:34.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:34 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1816695321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:05:35.363 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:05:34 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y[130308]: 2023-07-14T18:05:34.859+0000 7f756f19b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:05:35.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:35.355+0000 7f75f37fe700 1 -- 172.21.15.115:0/4150652548 <== mon.0 v2:172.21.15.115:3300/0 8 ==== mgrmap(e 15) v1 ==== 135660+0+0 (secure 0 0 0) 0x7f75f8021b10 con 0x7f76040fee80 2023-07-14T18:05:35.675 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:35 smithi187 ceph-mon[129317]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:35.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:35 smithi187 ceph-mon[129317]: Standby manager daemon y started 2023-07-14T18:05:35.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:35 smithi187 ceph-mon[129317]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T18:05:35.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:35 smithi187 ceph-mon[129317]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:05:35.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:35 smithi187 ceph-mon[129317]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T18:05:35.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:35 smithi187 ceph-mon[129317]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:05:35.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:35 smithi115 ceph-mon[127451]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:35.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:35 smithi115 ceph-mon[127451]: Standby manager daemon y started 2023-07-14T18:05:35.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:35 smithi115 ceph-mon[127451]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T18:05:35.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:35 smithi115 ceph-mon[127451]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:05:35.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:35 smithi115 ceph-mon[127451]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T18:05:35.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:35 smithi115 ceph-mon[127451]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:05:35.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:35 smithi150 ceph-mon[128853]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:35.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:35 smithi150 ceph-mon[128853]: Standby manager daemon y started 2023-07-14T18:05:35.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:35 smithi150 ceph-mon[128853]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T18:05:35.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:35 smithi150 ceph-mon[128853]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:05:35.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:35 smithi150 ceph-mon[128853]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T18:05:35.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:35 smithi150 ceph-mon[128853]: from='mgr.? 172.21.15.150:0/1447388170' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:05:36.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:36 smithi187 ceph-mon[129317]: mgrmap e15: x(active, since 66s), standbys: z, y 2023-07-14T18:05:36.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:36 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T18:05:36.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:36 smithi115 ceph-mon[127451]: mgrmap e15: x(active, since 66s), standbys: z, y 2023-07-14T18:05:36.768 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:36 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T18:05:36.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:36 smithi150 ceph-mon[128853]: mgrmap e15: x(active, since 66s), standbys: z, y 2023-07-14T18:05:36.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:36 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T18:05:37.675 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:37 smithi187 ceph-mon[129317]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:37.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:37 smithi115 ceph-mon[127451]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:37.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:37 smithi150 ceph-mon[128853]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:40.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:39 smithi187 ceph-mon[129317]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:40.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:39 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T18:05:40.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:39 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:40.273 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:39 smithi115 ceph-mon[127451]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:40.273 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:39 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T18:05:40.274 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:39 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:40.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:39 smithi150 ceph-mon[128853]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:40.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:39 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T18:05:40.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:39 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:41.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:40 smithi187 ceph-mon[129317]: Deploying daemon osd.0 on smithi115 2023-07-14T18:05:41.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:40 smithi115 ceph-mon[127451]: Deploying daemon osd.0 on smithi115 2023-07-14T18:05:41.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:40 smithi150 ceph-mon[128853]: Deploying daemon osd.0 on smithi115 2023-07-14T18:05:42.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:41 smithi187 ceph-mon[129317]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:42.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:41 smithi115 ceph-mon[127451]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:42.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:41 smithi150 ceph-mon[128853]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:43.674 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:43 smithi115 ceph-mon[127451]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:43.675 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:43 smithi187 ceph-mon[129317]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:43.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:43 smithi150 ceph-mon[128853]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:45.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:44 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:45.098 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:44 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:45.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:44 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:45.113 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:44 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:45.113 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:44 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:45.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:44 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:45.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:44 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:45.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:44 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:45.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:44 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:45.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:45 smithi115 ceph-mon[127451]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:45.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:45.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:45.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:45.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:45.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:46.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:45 smithi150 ceph-mon[128853]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:46.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:46.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:46.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:46.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:46.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:46.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:45 smithi187 ceph-mon[129317]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:46.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:46.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:46.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:46.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:46.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.536+0000 7f75f37fe700 1 -- 172.21.15.115:0/4150652548 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7604060a80 con 0x7f75ec0389e0 2023-07-14T18:05:46.568 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 0 on host 'smithi115' 2023-07-14T18:05:46.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75ec0389e0 msgr2=0x7f75ec03ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:46.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75ec0389e0 0x7f75ec03ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f75f4006010 tx=0x7f75f4005960 comp rx=0 tx=0).stop 2023-07-14T18:05:46.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fee80 msgr2=0x7f7604198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:46.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fee80 0x7f7604198020 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f75f800e9f0 tx=0x7f75f800ed00 comp rx=0 tx=0).stop 2023-07-14T18:05:46.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 shutdown_connections 2023-07-14T18:05:46.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75ec0389e0 0x7f75ec03ae90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:46.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76040ff830 0x7f760419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:46.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76040fe530 0x7f7604197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:46.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 --2- 172.21.15.115:0/4150652548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f76040fee80 0x7f7604198020 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:46.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.538+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 >> 172.21.15.115:0/4150652548 conn(0x7f76040fa1e0 msgr2=0x7f76040fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:46.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.539+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 shutdown_connections 2023-07-14T18:05:46.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:05:46.539+0000 7f7608e0b700 1 -- 172.21.15.115:0/4150652548 wait complete. 2023-07-14T18:05:47.150 DEBUG:teuthology.orchestra.run.smithi115:osd.0> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.0.service 2023-07-14T18:05:47.152 INFO:tasks.cephadm:Deploying osd.1 on smithi150 with /dev/vg_nvme/lv_4... 2023-07-14T18:05:47.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T18:05:47.183 INFO:journalctl@ceph.osd.0.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:05:47.714 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:05:47.811 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:05:47 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-0[140360]: 2023-07-14T18:05:47.696+0000 7efcd7a91640 -1 osd.0 0 log_to_monitors true 2023-07-14T18:05:47.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:47.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:47.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:47.812 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:47 smithi115 ceph-mon[127451]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:47.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:47.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:47.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:47.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:47 smithi150 ceph-mon[128853]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:47.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:47.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:47.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:05:47.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:47 smithi187 ceph-mon[129317]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:48.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:48 smithi115 ceph-mon[127451]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T18:05:48.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:48 smithi150 ceph-mon[128853]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T18:05:48.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:48 smithi187 ceph-mon[129317]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T18:05:49.089 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T18:05:49.089 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T18:05:49.089 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-07-14T18:05:49.089 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-07-14T18:05:49.090 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110656 s, 948 MB/s 2023-07-14T18:05:49.090 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-07-14T18:05:49.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2023-07-14T18:05:49.848 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-0[140360]: 2023-07-14T18:05:49.512+0000 7efccd109700 -1 osd.0 0 waiting for initial osdmap 2023-07-14T18:05:49.849 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-0[140360]: 2023-07-14T18:05:49.519+0000 7efcc76fc700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:05:49.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T18:05:49.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T18:05:49.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:49.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:05:49.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:49.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:05:49.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:49.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:49.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T18:05:49.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T18:05:49.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:49.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:05:49.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:49.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:05:49.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:49.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:49.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T18:05:49.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T18:05:49.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:49.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:05:49.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:49.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:05:49.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:49.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:49.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:05:49.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:05:50.017 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:05:50.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:50 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi115 2023-07-14T18:05:50.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:50 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi115 to 17168M 2023-07-14T18:05:50.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:50 smithi115 ceph-mon[127451]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:05:50.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:50 smithi115 ceph-mon[127451]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T18:05:50.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:50.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:50.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:50 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi115 2023-07-14T18:05:50.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:50 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi115 to 17168M 2023-07-14T18:05:50.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:50 smithi150 ceph-mon[128853]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:05:50.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:50 smithi150 ceph-mon[128853]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T18:05:50.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:50.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:50.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:50 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi115 2023-07-14T18:05:50.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:50 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi115 to 17168M 2023-07-14T18:05:50.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:50 smithi187 ceph-mon[129317]: from='osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:05:50.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:50 smithi187 ceph-mon[129317]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T18:05:50.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:50.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:51.191 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 -- 172.21.15.150:0/181239100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c0febb0 msgr2=0x7f2e1c0fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:51.191 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 --2- 172.21.15.150:0/181239100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c0febb0 0x7f2e1c0fefc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009990 tx=0x7f2e0c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:05:51.191 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 -- 172.21.15.150:0/181239100 shutdown_connections 2023-07-14T18:05:51.191 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 --2- 172.21.15.150:0/181239100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e1c100740 0x7f2e1c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:51.192 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 --2- 172.21.15.150:0/181239100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c0febb0 0x7f2e1c0fefc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:51.192 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 --2- 172.21.15.150:0/181239100 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e1c0ffdb0 0x7f2e1c100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:51.192 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 -- 172.21.15.150:0/181239100 >> 172.21.15.150:0/181239100 conn(0x7f2e1c0fa1e0 msgr2=0x7f2e1c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:05:51.192 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 -- 172.21.15.150:0/181239100 shutdown_connections 2023-07-14T18:05:51.192 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 -- 172.21.15.150:0/181239100 wait complete. 2023-07-14T18:05:51.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 Processor -- start 2023-07-14T18:05:51.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.160+0000 7f2e23715700 1 -- start start 2023-07-14T18:05:51.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e23715700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e1c0febb0 0x7f2e1c068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:51.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e23715700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e1c0ffdb0 0x7f2e1c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:51.194 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e23715700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c100740 0x7f2e1c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:51.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e23715700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c19d5a0 con 0x7f2e1c0ffdb0 2023-07-14T18:05:51.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e23715700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c19d710 con 0x7f2e1c100740 2023-07-14T18:05:51.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e23715700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c19d8d0 con 0x7f2e1c0febb0 2023-07-14T18:05:51.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e21cb2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c100740 0x7f2e1c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:51.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e21cb2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c100740 0x7f2e1c19ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53480/0 (socket says 172.21.15.150:53480) 2023-07-14T18:05:51.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e21cb2700 1 -- 172.21.15.150:0/3756786314 learned_addr learned my addr 172.21.15.150:0/3756786314 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-07-14T18:05:51.195 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e214b1700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e1c0febb0 0x7f2e1c068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:51.196 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e20cb0700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e1c0ffdb0 0x7f2e1c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:51.196 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e21cb2700 1 -- 172.21.15.150:0/3756786314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e1c0febb0 msgr2=0x7f2e1c068cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:51.196 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e21cb2700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e1c0febb0 0x7f2e1c068cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:51.196 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e21cb2700 1 -- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e1c0ffdb0 msgr2=0x7f2e1c19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:05:51.196 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e21cb2700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e1c0ffdb0 0x7f2e1c19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:05:51.196 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e21cb2700 1 -- 172.21.15.150:0/3756786314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c0095f0 con 0x7f2e1c100740 2023-07-14T18:05:51.196 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e20cb0700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e1c0ffdb0 0x7f2e1c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:05:51.197 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e214b1700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e1c0febb0 0x7f2e1c068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:05:51.197 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e21cb2700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c100740 0x7f2e1c19ce00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400b970 tx=0x7f2e1400bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:51.197 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e127fc700 1 -- 172.21.15.150:0/3756786314 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1400c9b0 con 0x7f2e1c100740 2023-07-14T18:05:51.197 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e127fc700 1 -- 172.21.15.150:0/3756786314 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2e14014440 con 0x7f2e1c100740 2023-07-14T18:05:51.197 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1c19dbb0 con 0x7f2e1c100740 2023-07-14T18:05:51.198 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e127fc700 1 -- 172.21.15.150:0/3756786314 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e140075d0 con 0x7f2e1c100740 2023-07-14T18:05:51.198 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.161+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1c19e0b0 con 0x7f2e1c100740 2023-07-14T18:05:51.198 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.162+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1c10a320 con 0x7f2e1c100740 2023-07-14T18:05:51.199 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.164+0000 7f2e127fc700 1 -- 172.21.15.150:0/3756786314 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 15) v1 ==== 135660+0+0 (secure 0 0 0) 0x7f2e140145b0 con 0x7f2e1c100740 2023-07-14T18:05:51.200 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.165+0000 7f2e127fc700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2e080a1140 0x7f2e080a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:05:51.200 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.165+0000 7f2e127fc700 1 -- 172.21.15.150:0/3756786314 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f2e14006e40 con 0x7f2e1c100740 2023-07-14T18:05:51.200 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.165+0000 7f2e214b1700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2e080a1140 0x7f2e080a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:05:51.200 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.165+0000 7f2e214b1700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2e080a1140 0x7f2e080a35f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00b5c0 tx=0x7f2e0c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:05:51.201 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.168+0000 7f2e127fc700 1 -- 172.21.15.150:0/3756786314 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2e140cae60 con 0x7f2e1c100740 2023-07-14T18:05:51.392 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:05:51.361+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e1c060a80 con 0x7f2e080a1140 2023-07-14T18:05:51.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:05:51.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:05:51.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:51.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] boot 2023-07-14T18:05:51.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T18:05:51.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:51.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:51.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:05:51.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:05:51.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:51.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:05:51.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:05:51.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:51.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] boot 2023-07-14T18:05:51.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T18:05:51.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:51.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:51.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:05:51.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:05:51.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:51.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:05:51.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:05:51.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:51.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: osd.0 [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] boot 2023-07-14T18:05:51.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T18:05:51.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:05:51.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:05:51.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:05:51.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:05:51.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:05:52.731 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:52 smithi150 ceph-mon[128853]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:52.732 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:52 smithi150 ceph-mon[128853]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T18:05:52.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:52 smithi115 ceph-mon[127451]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:52.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:52 smithi115 ceph-mon[127451]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T18:05:52.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:52 smithi187 ceph-mon[129317]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:05:52.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:52 smithi187 ceph-mon[129317]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T18:05:53.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:53 smithi115 ceph-mon[127451]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:05:53.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:53 smithi150 ceph-mon[128853]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:05:53.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:53 smithi187 ceph-mon[129317]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:05:54.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:54 smithi115 ceph-mon[127451]: from='client.? 172.21.15.150:0/3092929573' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffda5af6-01b2-4a92-863f-7f6518223778"}]: dispatch 2023-07-14T18:05:54.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:54 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffda5af6-01b2-4a92-863f-7f6518223778"}]: dispatch 2023-07-14T18:05:54.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:54 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffda5af6-01b2-4a92-863f-7f6518223778"}]': finished 2023-07-14T18:05:54.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:54 smithi115 ceph-mon[127451]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T18:05:54.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:05:54.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:54 smithi115 ceph-mon[127451]: from='client.? 172.21.15.150:0/1009994329' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:05:54.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:54 smithi150 ceph-mon[128853]: from='client.? 172.21.15.150:0/3092929573' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffda5af6-01b2-4a92-863f-7f6518223778"}]: dispatch 2023-07-14T18:05:54.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:54 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffda5af6-01b2-4a92-863f-7f6518223778"}]: dispatch 2023-07-14T18:05:54.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:54 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffda5af6-01b2-4a92-863f-7f6518223778"}]': finished 2023-07-14T18:05:54.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:54 smithi150 ceph-mon[128853]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T18:05:54.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:05:54.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:54 smithi150 ceph-mon[128853]: from='client.? 172.21.15.150:0/1009994329' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:05:54.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:54 smithi187 ceph-mon[129317]: from='client.? 172.21.15.150:0/3092929573' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffda5af6-01b2-4a92-863f-7f6518223778"}]: dispatch 2023-07-14T18:05:54.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:54 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffda5af6-01b2-4a92-863f-7f6518223778"}]: dispatch 2023-07-14T18:05:54.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:54 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffda5af6-01b2-4a92-863f-7f6518223778"}]': finished 2023-07-14T18:05:54.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:54 smithi187 ceph-mon[129317]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T18:05:54.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:05:54.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:54 smithi187 ceph-mon[129317]: from='client.? 172.21.15.150:0/1009994329' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:05:55.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:55 smithi115 ceph-mon[127451]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:05:55.863 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:55 smithi150 ceph-mon[128853]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:05:55.926 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:55 smithi187 ceph-mon[129317]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:05:58.113 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:57 smithi150 ceph-mon[128853]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:05:58.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:57 smithi187 ceph-mon[129317]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:05:58.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:57 smithi115 ceph-mon[127451]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:00.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:05:59 smithi187 ceph-mon[129317]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:00.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:05:59 smithi115 ceph-mon[127451]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:00.363 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:05:59 smithi150 ceph-mon[128853]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:01.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:00 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T18:06:01.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:00 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:01.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:00 smithi150 ceph-mon[128853]: Deploying daemon osd.1 on smithi150 2023-07-14T18:06:01.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:00 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T18:06:01.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:00 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:01.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:00 smithi187 ceph-mon[129317]: Deploying daemon osd.1 on smithi150 2023-07-14T18:06:01.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:00 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T18:06:01.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:00 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:01.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:00 smithi115 ceph-mon[127451]: Deploying daemon osd.1 on smithi150 2023-07-14T18:06:02.176 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:01 smithi187 ceph-mon[129317]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:02.253 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:01 smithi150 ceph-mon[128853]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:02.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:01 smithi115 ceph-mon[127451]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:03.614 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:03 smithi150 ceph-mon[128853]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:03.676 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:03 smithi187 ceph-mon[129317]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:03.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:03 smithi115 ceph-mon[127451]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:05.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:05.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:05.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:05.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:05.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:05.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:05.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:05.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:05.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:06.334 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:06 smithi150 ceph-mon[128853]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:06.348 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:06 smithi115 ceph-mon[127451]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:06.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:06 smithi187 ceph-mon[129317]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:06.656 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.624+0000 7f2e127fc700 1 -- 172.21.15.150:0/3756786314 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2e1c060a80 con 0x7f2e080a1140 2023-07-14T18:06:06.657 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 1 on host 'smithi150' 2023-07-14T18:06:06.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.626+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2e080a1140 msgr2=0x7f2e080a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:06.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.626+0000 7f2e23715700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2e080a1140 0x7f2e080a35f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00b5c0 tx=0x7f2e0c005fb0 comp rx=0 tx=0).stop 2023-07-14T18:06:06.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.626+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c100740 msgr2=0x7f2e1c19ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:06.658 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.626+0000 7f2e23715700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c100740 0x7f2e1c19ce00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400b970 tx=0x7f2e1400bd30 comp rx=0 tx=0).stop 2023-07-14T18:06:06.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.627+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 shutdown_connections 2023-07-14T18:06:06.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.627+0000 7f2e23715700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2e080a1140 0x7f2e080a35f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:06.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.627+0000 7f2e23715700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e1c100740 0x7f2e1c19ce00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:06.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.627+0000 7f2e23715700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2e1c0febb0 0x7f2e1c068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:06.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.627+0000 7f2e23715700 1 --2- 172.21.15.150:0/3756786314 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e1c0ffdb0 0x7f2e1c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:06.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.627+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 >> 172.21.15.150:0/3756786314 conn(0x7f2e1c0fa1e0 msgr2=0x7f2e1c0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:06:06.659 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.627+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 shutdown_connections 2023-07-14T18:06:06.660 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:06.627+0000 7f2e23715700 1 -- 172.21.15.150:0/3756786314 wait complete. 2023-07-14T18:06:06.991 DEBUG:teuthology.orchestra.run.smithi150:osd.1> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.1.service 2023-07-14T18:06:06.993 INFO:tasks.cephadm:Deploying osd.2 on smithi187 with /dev/vg_nvme/lv_4... 2023-07-14T18:06:06.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T18:06:07.024 INFO:journalctl@ceph.osd.1.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:06:07.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.426 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:07.553 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:06:08.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:08 smithi115 ceph-mon[127451]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:08.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:08 smithi115 ceph-mon[127451]: from='osd.1 [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:06:08.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:08 smithi115 ceph-mon[127451]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:06:08.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:08 smithi150 ceph-mon[128853]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:08.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:08 smithi150 ceph-mon[128853]: from='osd.1 [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:06:08.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:08 smithi150 ceph-mon[128853]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:06:08.365 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:07 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-1[133833]: 2023-07-14T18:06:07.982+0000 7fa8c3f62640 -1 osd.1 0 log_to_monitors true 2023-07-14T18:06:08.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:08 smithi187 ceph-mon[129317]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:08.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:08 smithi187 ceph-mon[129317]: from='osd.1 [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:06:08.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:08 smithi187 ceph-mon[129317]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:06:09.037 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T18:06:09.037 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T18:06:09.038 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2023-07-14T18:06:09.038 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2023-07-14T18:06:09.038 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.010928 s, 960 MB/s 2023-07-14T18:06:09.038 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2023-07-14T18:06:09.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi187:vg_nvme/lv_4 2023-07-14T18:06:09.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T18:06:09.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T18:06:09.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:09.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='osd.1 [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:06:09.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:06:09.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:09.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi150 2023-07-14T18:06:09.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:09.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:09.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:09.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi150 to 17168M 2023-07-14T18:06:09.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:09.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:09.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:09.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:09.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:09.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:09.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:09.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.055 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:06:10.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T18:06:10.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T18:06:10.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:10.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='osd.1 [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:06:10.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:06:10.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:10.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi150 2023-07-14T18:06:10.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:10.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi150 to 17168M 2023-07-14T18:06:10.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:10.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:10.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:10.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:10.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:10.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.114 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-1[133833]: 2023-07-14T18:06:09.608+0000 7fa8b95da700 -1 osd.1 0 waiting for initial osdmap 2023-07-14T18:06:10.114 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-1[133833]: 2023-07-14T18:06:09.611+0000 7fa8b53d0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:06:10.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T18:06:10.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T18:06:10.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:10.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='osd.1 [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:06:10.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:06:10.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:06:10.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi150 2023-07-14T18:06:10.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:10.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi150 to 17168M 2023-07-14T18:06:10.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:10.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:10.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:10.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:10.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:10.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:10.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:10 smithi187 ceph-mon[129317]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:06:10.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:10 smithi187 ceph-mon[129317]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T18:06:10.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:10.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:11.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:10 smithi115 ceph-mon[127451]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:06:11.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:10 smithi115 ceph-mon[127451]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T18:06:11.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:11.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:11.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:10 smithi150 ceph-mon[128853]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:06:11.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:10 smithi150 ceph-mon[128853]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T18:06:11.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:11.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:11.282 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.250+0000 7faf1e71c700 1 -- 172.21.15.187:0/1805194897 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf180ffcd0 msgr2=0x7faf181000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:11.282 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.250+0000 7faf1e71c700 1 --2- 172.21.15.187:0/1805194897 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf180ffcd0 0x7faf181000e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faf08009990 tx=0x7faf08009ca0 comp rx=0 tx=0).stop 2023-07-14T18:06:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.250+0000 7faf1e71c700 1 -- 172.21.15.187:0/1805194897 shutdown_connections 2023-07-14T18:06:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.250+0000 7faf1e71c700 1 --2- 172.21.15.187:0/1805194897 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf18101920 0x7faf181081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.250+0000 7faf1e71c700 1 --2- 172.21.15.187:0/1805194897 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf180ffcd0 0x7faf181000e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.250+0000 7faf1e71c700 1 --2- 172.21.15.187:0/1805194897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf18100f70 0x7faf181013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.250+0000 7faf1e71c700 1 -- 172.21.15.187:0/1805194897 >> 172.21.15.187:0/1805194897 conn(0x7faf180fb320 msgr2=0x7faf180fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:06:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.250+0000 7faf1e71c700 1 -- 172.21.15.187:0/1805194897 shutdown_connections 2023-07-14T18:06:11.283 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.250+0000 7faf1e71c700 1 -- 172.21.15.187:0/1805194897 wait complete. 2023-07-14T18:06:11.285 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf1e71c700 1 Processor -- start 2023-07-14T18:06:11.285 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf1e71c700 1 -- start start 2023-07-14T18:06:11.285 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf1e71c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf180ffcd0 0x7faf180743e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf1e71c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf18100f70 0x7faf18072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf1e71c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf18101920 0x7faf18072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf1e71c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf180736f0 con 0x7faf180ffcd0 2023-07-14T18:06:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf1e71c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf18073860 con 0x7faf18101920 2023-07-14T18:06:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf1e71c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1819f910 con 0x7faf18100f70 2023-07-14T18:06:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf177fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf18100f70 0x7faf18072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:11.286 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf177fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf18100f70 0x7faf18072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:58474/0 (socket says 172.21.15.187:58474) 2023-07-14T18:06:11.287 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf177fe700 1 -- 172.21.15.187:0/941531972 learned_addr learned my addr 172.21.15.187:0/941531972 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:06:11.287 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf17fff700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf180ffcd0 0x7faf180743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:11.287 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf1ccb9700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf18101920 0x7faf18072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:11.287 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.251+0000 7faf177fe700 1 -- 172.21.15.187:0/941531972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf18101920 msgr2=0x7faf18072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:11.287 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf177fe700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf18101920 0x7faf18072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:11.287 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf177fe700 1 -- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf180ffcd0 msgr2=0x7faf180743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf177fe700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf180ffcd0 0x7faf180743e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf177fe700 1 -- 172.21.15.187:0/941531972 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf080095f0 con 0x7faf18100f70 2023-07-14T18:06:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf177fe700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf18100f70 0x7faf18072a30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faf0000d950 tx=0x7faf0000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:06:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf157fa700 1 -- 172.21.15.187:0/941531972 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0000f890 con 0x7faf18100f70 2023-07-14T18:06:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf157fa700 1 -- 172.21.15.187:0/941531972 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faf0000fe80 con 0x7faf18100f70 2023-07-14T18:06:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf157fa700 1 -- 172.21.15.187:0/941531972 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0000e3b0 con 0x7faf18100f70 2023-07-14T18:06:11.288 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1819fbf0 con 0x7faf18100f70 2023-07-14T18:06:11.289 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.252+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf181a00a0 con 0x7faf18100f70 2023-07-14T18:06:11.289 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.253+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1810b350 con 0x7faf18100f70 2023-07-14T18:06:11.289 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.254+0000 7faf157fa700 1 -- 172.21.15.187:0/941531972 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 15) v1 ==== 135660+0+0 (secure 0 0 0) 0x7faf00010460 con 0x7faf18100f70 2023-07-14T18:06:11.290 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.254+0000 7faf157fa700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faf040a10c0 0x7faf040a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:11.290 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.255+0000 7faf157fa700 1 -- 172.21.15.187:0/941531972 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7faf000caec0 con 0x7faf18100f70 2023-07-14T18:06:11.290 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.255+0000 7faf17fff700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faf040a10c0 0x7faf040a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:11.290 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.255+0000 7faf17fff700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faf040a10c0 0x7faf040a3570 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf08006010 tx=0x7faf08005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:06:11.290 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.259+0000 7faf157fa700 1 -- 172.21.15.187:0/941531972 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faf00095dc0 con 0x7faf18100f70 2023-07-14T18:06:11.482 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:11.450+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7faf18060a80 con 0x7faf040a10c0 2023-07-14T18:06:11.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:11 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:06:11.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:11 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:06:11.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:11 smithi187 ceph-mon[129317]: osd.1 [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] boot 2023-07-14T18:06:11.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:11 smithi187 ceph-mon[129317]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T18:06:11.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:11.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:11 smithi187 ceph-mon[129317]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:11.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:06:11.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:06:11.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:12.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:11 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:06:12.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:11 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:06:12.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:11 smithi115 ceph-mon[127451]: osd.1 [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] boot 2023-07-14T18:06:12.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:11 smithi115 ceph-mon[127451]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T18:06:12.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:12.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:11 smithi115 ceph-mon[127451]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:12.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:06:12.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:06:12.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:12.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:11 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:06:12.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:11 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:06:12.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:11 smithi150 ceph-mon[128853]: osd.1 [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] boot 2023-07-14T18:06:12.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:11 smithi150 ceph-mon[128853]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T18:06:12.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:06:12.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:11 smithi150 ceph-mon[128853]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:12.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:06:12.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:06:12.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:12.837 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:12 smithi187 ceph-mon[129317]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:06:12.838 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:12 smithi187 ceph-mon[129317]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T18:06:13.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:12 smithi115 ceph-mon[127451]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:06:13.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:12 smithi115 ceph-mon[127451]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T18:06:13.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:12 smithi150 ceph-mon[128853]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:06:13.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:12 smithi150 ceph-mon[128853]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T18:06:13.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:13 smithi187 ceph-mon[129317]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:14.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:13 smithi115 ceph-mon[127451]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:14.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:13 smithi150 ceph-mon[128853]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:14.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:14 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/1348450025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83a08d79-e0ac-47ea-a061-5b92c4a33d3e"}]: dispatch 2023-07-14T18:06:14.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:14 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83a08d79-e0ac-47ea-a061-5b92c4a33d3e"}]: dispatch 2023-07-14T18:06:14.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:14 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83a08d79-e0ac-47ea-a061-5b92c4a33d3e"}]': finished 2023-07-14T18:06:14.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:14 smithi187 ceph-mon[129317]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T18:06:14.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:14.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:14 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/3691947550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:06:15.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:14 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/1348450025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83a08d79-e0ac-47ea-a061-5b92c4a33d3e"}]: dispatch 2023-07-14T18:06:15.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:14 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83a08d79-e0ac-47ea-a061-5b92c4a33d3e"}]: dispatch 2023-07-14T18:06:15.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:14 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83a08d79-e0ac-47ea-a061-5b92c4a33d3e"}]': finished 2023-07-14T18:06:15.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:14 smithi115 ceph-mon[127451]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T18:06:15.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:15.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:14 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/3691947550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:06:15.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:14 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/1348450025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83a08d79-e0ac-47ea-a061-5b92c4a33d3e"}]: dispatch 2023-07-14T18:06:15.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:14 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83a08d79-e0ac-47ea-a061-5b92c4a33d3e"}]: dispatch 2023-07-14T18:06:15.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:14 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83a08d79-e0ac-47ea-a061-5b92c4a33d3e"}]': finished 2023-07-14T18:06:15.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:14 smithi150 ceph-mon[128853]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T18:06:15.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:15.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:14 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/3691947550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:06:15.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:15 smithi187 ceph-mon[129317]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:16.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:15 smithi115 ceph-mon[127451]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:16.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:15 smithi150 ceph-mon[128853]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:18.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:17 smithi187 ceph-mon[129317]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:18.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:17 smithi115 ceph-mon[127451]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:18.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:17 smithi150 ceph-mon[128853]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:20.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:19 smithi187 ceph-mon[129317]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:20.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:19 smithi115 ceph-mon[127451]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:20.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:19 smithi150 ceph-mon[128853]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:21.059 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T18:06:21.059 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:20 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:21.059 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:20 smithi187 ceph-mon[129317]: Deploying daemon osd.2 on smithi187 2023-07-14T18:06:21.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T18:06:21.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:20 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:21.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:20 smithi115 ceph-mon[127451]: Deploying daemon osd.2 on smithi187 2023-07-14T18:06:21.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T18:06:21.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:20 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:21.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:20 smithi150 ceph-mon[128853]: Deploying daemon osd.2 on smithi187 2023-07-14T18:06:22.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:21 smithi187 ceph-mon[129317]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:22.349 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:21 smithi115 ceph-mon[127451]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:22.364 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:21 smithi150 ceph-mon[128853]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:23.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:23 smithi187 ceph-mon[129317]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:23.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:23 smithi115 ceph-mon[127451]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:23.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:23 smithi150 ceph-mon[128853]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:24.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:24 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:24.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:24 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:24.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:24 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:24.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:24 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:24.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:24 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:24.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:24 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:24.883 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:24 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:24.883 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:24 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:24.883 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:24 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:25.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:25 smithi115 ceph-mon[127451]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:25.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:25 smithi150 ceph-mon[128853]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:25.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:25 smithi187 ceph-mon[129317]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:27.038 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 2 on host 'smithi187' 2023-07-14T18:06:27.038 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.005+0000 7faf157fa700 1 -- 172.21.15.187:0/941531972 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faf18060a80 con 0x7faf040a10c0 2023-07-14T18:06:27.040 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faf040a10c0 msgr2=0x7faf040a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:27.040 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faf040a10c0 0x7faf040a3570 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf08006010 tx=0x7faf08005960 comp rx=0 tx=0).stop 2023-07-14T18:06:27.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf18100f70 msgr2=0x7faf18072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:27.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf18100f70 0x7faf18072a30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faf0000d950 tx=0x7faf0000dc60 comp rx=0 tx=0).stop 2023-07-14T18:06:27.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 shutdown_connections 2023-07-14T18:06:27.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faf040a10c0 0x7faf040a3570 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:27.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf18101920 0x7faf18072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:27.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faf180ffcd0 0x7faf180743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:27.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 --2- 172.21.15.187:0/941531972 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf18100f70 0x7faf18072a30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:27.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 >> 172.21.15.187:0/941531972 conn(0x7faf180fb320 msgr2=0x7faf180fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:06:27.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 shutdown_connections 2023-07-14T18:06:27.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:06:27.008+0000 7faf1e71c700 1 -- 172.21.15.187:0/941531972 wait complete. 2023-07-14T18:06:27.440 DEBUG:teuthology.orchestra.run.smithi187:osd.2> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.2.service 2023-07-14T18:06:27.442 INFO:tasks.cephadm:Deploying osd.3 on smithi115 with /dev/vg_nvme/lv_3... 2023-07-14T18:06:27.442 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T18:06:27.472 INFO:journalctl@ceph.osd.2.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:06:27.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:27.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:27.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:27 smithi187 ceph-mon[129317]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:27.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:27 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:27.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:27.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:27.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:27 smithi115 ceph-mon[127451]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:27.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:27 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:27.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:27.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:27.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:27 smithi150 ceph-mon[128853]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:27.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:27.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:27 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:28.005 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:06:28.495 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-2[135131]: 2023-07-14T18:06:28.409+0000 7fc6a808b640 -1 osd.2 0 log_to_monitors true 2023-07-14T18:06:28.725 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:28 smithi115 ceph-mon[127451]: from='osd.2 [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:06:28.726 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:28 smithi115 ceph-mon[127451]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:06:28.768 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:28 smithi187 ceph-mon[129317]: from='osd.2 [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:06:28.769 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:28 smithi187 ceph-mon[129317]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:06:28.864 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:28 smithi150 ceph-mon[128853]: from='osd.2 [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:06:28.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:28 smithi150 ceph-mon[128853]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:06:29.428 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T18:06:29.428 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T18:06:29.428 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2023-07-14T18:06:29.428 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2023-07-14T18:06:29.429 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110433 s, 950 MB/s 2023-07-14T18:06:29.429 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2023-07-14T18:06:29.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:29 smithi187 ceph-mon[129317]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T18:06:29.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:29 smithi187 ceph-mon[129317]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T18:06:29.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:29.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:29 smithi187 ceph-mon[129317]: from='osd.2 [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:06:29.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:29 smithi187 ceph-mon[129317]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:06:29.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:29 smithi187 ceph-mon[129317]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:29.679 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-2[135131]: 2023-07-14T18:06:29.484+0000 7fc69d703700 -1 osd.2 0 waiting for initial osdmap 2023-07-14T18:06:29.679 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-2[135131]: 2023-07-14T18:06:29.489+0000 7fc696cf4700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:06:29.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:29 smithi115 ceph-mon[127451]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T18:06:29.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:29 smithi115 ceph-mon[127451]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T18:06:29.848 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:29.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:29 smithi115 ceph-mon[127451]: from='osd.2 [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:06:29.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:29 smithi115 ceph-mon[127451]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:06:29.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:29 smithi115 ceph-mon[127451]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:29.850 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_3 2023-07-14T18:06:29.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:29 smithi150 ceph-mon[128853]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T18:06:29.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:29 smithi150 ceph-mon[128853]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T18:06:29.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:29.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:29 smithi150 ceph-mon[128853]: from='osd.2 [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:06:29.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:29 smithi150 ceph-mon[128853]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:06:29.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:29 smithi150 ceph-mon[128853]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:06:30.402 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:06:30.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:06:30.776 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T18:06:30.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:30.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:30.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi187 2023-07-14T18:06:30.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:30.777 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi187 to 17168M 2023-07-14T18:06:30.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:30.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:30.778 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:06:30.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T18:06:30.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:30.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:30.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi187 2023-07-14T18:06:30.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:30.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi187 to 17168M 2023-07-14T18:06:30.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:30.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:30.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:06:30.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T18:06:30.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:30.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:30.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi187 2023-07-14T18:06:30.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:30.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi187 to 17168M 2023-07-14T18:06:30.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:30.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:30.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:30.929 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:31.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.443+0000 7f75de0a6700 1 -- 172.21.15.115:0/1287693317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d80fdfa0 msgr2=0x7f75d80fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:31.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.443+0000 7f75de0a6700 1 --2- 172.21.15.115:0/1287693317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d80fdfa0 0x7f75d80fe3b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0099c0 tx=0x7f75cc009cd0 comp rx=0 tx=0).stop 2023-07-14T18:06:31.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.443+0000 7f75de0a6700 1 -- 172.21.15.115:0/1287693317 shutdown_connections 2023-07-14T18:06:31.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.443+0000 7f75de0a6700 1 --2- 172.21.15.115:0/1287693317 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d81019b0 0x7f75d8108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:31.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.443+0000 7f75de0a6700 1 --2- 172.21.15.115:0/1287693317 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d80fdfa0 0x7f75d80fe3b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:31.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.443+0000 7f75de0a6700 1 --2- 172.21.15.115:0/1287693317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8101020 0x7f75d8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:31.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.443+0000 7f75de0a6700 1 -- 172.21.15.115:0/1287693317 >> 172.21.15.115:0/1287693317 conn(0x7f75d80f9990 msgr2=0x7f75d80fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:06:31.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.443+0000 7f75de0a6700 1 -- 172.21.15.115:0/1287693317 shutdown_connections 2023-07-14T18:06:31.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.443+0000 7f75de0a6700 1 -- 172.21.15.115:0/1287693317 wait complete. 2023-07-14T18:06:31.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75de0a6700 1 Processor -- start 2023-07-14T18:06:31.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75de0a6700 1 -- start start 2023-07-14T18:06:31.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75de0a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d80fdfa0 0x7f75d8195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:31.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75de0a6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d8101020 0x7f75d8195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:31.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75de0a6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d81019b0 0x7f75d819cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:31.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75de0a6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d819d4b0 con 0x7f75d8101020 2023-07-14T18:06:31.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75de0a6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d819d5f0 con 0x7f75d81019b0 2023-07-14T18:06:31.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75de0a6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d819d780 con 0x7f75d80fdfa0 2023-07-14T18:06:31.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d8101020 0x7f75d8195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:31.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d8101020 0x7f75d8195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51722/0 (socket says 172.21.15.115:51722) 2023-07-14T18:06:31.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d6ffd700 1 -- 172.21.15.115:0/2365886057 learned_addr learned my addr 172.21.15.115:0/2365886057 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:06:31.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d77fe700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d80fdfa0 0x7f75d8195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:31.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d7fff700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d81019b0 0x7f75d819cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:31.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d6ffd700 1 -- 172.21.15.115:0/2365886057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d80fdfa0 msgr2=0x7f75d8195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:31.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d6ffd700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d80fdfa0 0x7f75d8195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:31.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d6ffd700 1 -- 172.21.15.115:0/2365886057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d81019b0 msgr2=0x7f75d819cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:31.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d6ffd700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d81019b0 0x7f75d819cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:31.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.444+0000 7f75d6ffd700 1 -- 172.21.15.115:0/2365886057 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc0095f0 con 0x7f75d8101020 2023-07-14T18:06:31.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.445+0000 7f75d6ffd700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d8101020 0x7f75d8195e60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f75c800d980 tx=0x7f75c800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:06:31.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.445+0000 7f75d4ff9700 1 -- 172.21.15.115:0/2365886057 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c80098d0 con 0x7f75d8101020 2023-07-14T18:06:31.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.445+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d819da60 con 0x7f75d8101020 2023-07-14T18:06:31.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.445+0000 7f75d4ff9700 1 -- 172.21.15.115:0/2365886057 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f75c8010460 con 0x7f75d8101020 2023-07-14T18:06:31.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.445+0000 7f75d4ff9700 1 -- 172.21.15.115:0/2365886057 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c800f620 con 0x7f75d8101020 2023-07-14T18:06:31.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.445+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d819dfb0 con 0x7f75d8101020 2023-07-14T18:06:31.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.446+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d8107450 con 0x7f75d8101020 2023-07-14T18:06:31.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.447+0000 7f75d4ff9700 1 -- 172.21.15.115:0/2365886057 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 15) v1 ==== 135660+0+0 (secure 0 0 0) 0x7f75c8009a30 con 0x7f75d8101020 2023-07-14T18:06:31.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.447+0000 7f75d4ff9700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75c00a1140 0x7f75c00a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:31.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.447+0000 7f75d77fe700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75c00a1140 0x7f75c00a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:31.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.447+0000 7f75d4ff9700 1 -- 172.21.15.115:0/2365886057 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f75c80cb0c0 con 0x7f75d8101020 2023-07-14T18:06:31.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.448+0000 7f75d77fe700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75c00a1140 0x7f75c00a35f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00c010 tx=0x7f75cc005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:06:31.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.451+0000 7f75d4ff9700 1 -- 172.21.15.115:0/2365886057 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f75c8095da0 con 0x7f75d8101020 2023-07-14T18:06:31.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:31.650+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f75d8060a80 con 0x7f75c00a1140 2023-07-14T18:06:31.849 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:31 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:06:31.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:31 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:06:31.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:31 smithi115 ceph-mon[127451]: osd.2 [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] boot 2023-07-14T18:06:31.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:31 smithi115 ceph-mon[127451]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T18:06:31.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:31.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:31 smithi115 ceph-mon[127451]: pgmap v69: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:31.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:06:31.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:31 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:06:31.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:31 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:06:31.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:31 smithi150 ceph-mon[128853]: osd.2 [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] boot 2023-07-14T18:06:31.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:31 smithi150 ceph-mon[128853]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T18:06:31.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:31.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:31 smithi150 ceph-mon[128853]: pgmap v69: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:31.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:06:31.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:31 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:06:31.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:31 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:06:31.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:31 smithi187 ceph-mon[129317]: osd.2 [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] boot 2023-07-14T18:06:31.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:31 smithi187 ceph-mon[129317]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T18:06:31.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:06:31.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:31 smithi187 ceph-mon[129317]: pgmap v69: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:31.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:06:32.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T18:06:32.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:32 smithi115 ceph-mon[127451]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T18:06:32.803 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:06:32.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:32 smithi115 ceph-mon[127451]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:06:32.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:06:32.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:06:32.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:32.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T18:06:32.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:32 smithi150 ceph-mon[128853]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T18:06:32.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:06:32.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:32 smithi150 ceph-mon[128853]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:06:32.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:06:32.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:06:32.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:32.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T18:06:32.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:32 smithi187 ceph-mon[129317]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T18:06:32.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:06:32.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:32 smithi187 ceph-mon[129317]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:06:32.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:06:32.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:06:32.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:33.099 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:06:32 smithi115 sudo[143457]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:06:33.100 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:06:32 smithi115 sudo[143457]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:33.100 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:06:32 smithi115 sudo[143457]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:33.100 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:06:32 smithi115 sudo[143457]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:33.418 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:06:33 smithi115 sudo[143463]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:06:33.419 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:06:33 smithi115 sudo[143463]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:33.419 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:06:33 smithi115 sudo[143463]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:33.419 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:06:33 smithi115 sudo[143463]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:33.664 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:33 smithi150 sudo[136219]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:06:33.664 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:33 smithi150 sudo[136219]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:33.664 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:33 smithi150 sudo[136219]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:33.664 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:33 smithi150 sudo[136219]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:33.665 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:06:33.665 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:33 smithi150 ceph-mon[128853]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T18:06:33.665 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:33 smithi150 ceph-mon[128853]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:33.665 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:33.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:06:33.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:33 smithi115 ceph-mon[127451]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T18:06:33.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:33 smithi115 ceph-mon[127451]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:33.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:33.927 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:06:33.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:33 smithi187 ceph-mon[129317]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T18:06:33.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:33 smithi187 ceph-mon[129317]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:33.928 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:34.115 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:33 smithi150 sudo[136222]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:06:34.115 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:33 smithi150 sudo[136222]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:34.115 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:33 smithi150 sudo[136222]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:34.115 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:06:33 smithi150 sudo[136222]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:34.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:34.143+0000 7f75d4ff9700 1 -- 172.21.15.115:0/2365886057 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f75c80f7780 con 0x7f75d8101020 2023-07-14T18:06:34.189 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:33 smithi187 sudo[137387]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:06:34.190 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:33 smithi187 sudo[137387]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:34.190 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:33 smithi187 sudo[137387]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:34.190 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:33 smithi187 sudo[137387]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:34.677 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:34 smithi187 sudo[137390]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:06:34.677 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:34 smithi187 sudo[137390]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:34.678 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:34 smithi187 sudo[137390]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:34.678 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:06:34 smithi187 sudo[137390]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:34.723 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 sudo[143564]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:06:34.724 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 sudo[143564]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:34.724 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 sudo[143564]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:34.724 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 sudo[143564]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:34.986 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 ceph-mon[129317]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T18:06:34.987 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:34.987 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/107565901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9796f71f-7cff-4ee0-ae32-545fb5fc886d"}]: dispatch 2023-07-14T18:06:34.987 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/107565901' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9796f71f-7cff-4ee0-ae32-545fb5fc886d"}]': finished 2023-07-14T18:06:34.987 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 ceph-mon[129317]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T18:06:34.987 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:34.987 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 ceph-mon[129317]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T18:06:34.987 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:34.988 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 ceph-mon[129317]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:06:34.988 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 sudo[137393]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:06:34.988 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 sudo[137393]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:34.988 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 sudo[137393]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:34.988 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:34 smithi187 sudo[137393]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:35.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 sudo[143585]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:06:35.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 sudo[143585]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:35.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 sudo[143585]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:35.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 sudo[143585]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:35.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 ceph-mon[127451]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T18:06:35.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:35.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/107565901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9796f71f-7cff-4ee0-ae32-545fb5fc886d"}]: dispatch 2023-07-14T18:06:35.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/107565901' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9796f71f-7cff-4ee0-ae32-545fb5fc886d"}]': finished 2023-07-14T18:06:35.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 ceph-mon[127451]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T18:06:35.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:35.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 ceph-mon[127451]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T18:06:35.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:35.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:34 smithi115 ceph-mon[127451]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:06:35.114 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:34 smithi150 ceph-mon[128853]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T18:06:35.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:34 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:35.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:34 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/107565901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9796f71f-7cff-4ee0-ae32-545fb5fc886d"}]: dispatch 2023-07-14T18:06:35.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:34 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/107565901' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9796f71f-7cff-4ee0-ae32-545fb5fc886d"}]': finished 2023-07-14T18:06:35.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:34 smithi150 ceph-mon[128853]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T18:06:35.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:34 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:35.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:34 smithi150 ceph-mon[128853]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T18:06:35.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:34 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:35.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:34 smithi150 ceph-mon[128853]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:06:35.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 sudo[137396]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:06:35.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 sudo[137396]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:35.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 sudo[137396]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:35.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 sudo[137396]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:35.717 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 sudo[136225]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:06:35.718 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 sudo[136225]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:35.718 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 sudo[136225]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:35.718 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 sudo[136225]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:36.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:06:36.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:06:36.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:06:36.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:06:36.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:06:36.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:06:36.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:06:36.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:06:36.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:36.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2650309972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:06:36.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:06:36.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:06:36.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:06:36.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:06:36.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:06:36.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:35 smithi115 ceph-mon[127451]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:06:36.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 sudo[136228]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:06:36.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 sudo[136228]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:06:36.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 sudo[136228]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:06:36.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 sudo[136228]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:06:36.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:06:36.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:06:36.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:06:36.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:06:36.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:06:36.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:06:36.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:06:36.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:06:36.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:36.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2650309972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:06:36.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:06:36.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:06:36.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:06:36.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:06:36.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:06:36.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:35 smithi150 ceph-mon[128853]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:06:36.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:06:36.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:06:36.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:06:36.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:06:36.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:06:36.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:06:36.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:06:36.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:06:36.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:36.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2650309972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:06:36.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:06:36.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:06:36.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:06:36.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:06:36.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:06:36.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:35 smithi187 ceph-mon[129317]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:06:38.169 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:37 smithi115 ceph-mon[127451]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:38.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:37 smithi187 ceph-mon[129317]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:38.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:37 smithi150 ceph-mon[128853]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:40.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:39 smithi187 ceph-mon[129317]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:40.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:39 smithi115 ceph-mon[127451]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:40.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:39 smithi150 ceph-mon[128853]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:41.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:40 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T18:06:41.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:40 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:41.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:40 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T18:06:41.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:40 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:41.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:40 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T18:06:41.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:40 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:42.099 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:41 smithi115 ceph-mon[127451]: Deploying daemon osd.3 on smithi115 2023-07-14T18:06:42.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:41 smithi115 ceph-mon[127451]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-14T18:06:42.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:41 smithi187 ceph-mon[129317]: Deploying daemon osd.3 on smithi115 2023-07-14T18:06:42.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:41 smithi187 ceph-mon[129317]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-14T18:06:42.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:41 smithi150 ceph-mon[128853]: Deploying daemon osd.3 on smithi115 2023-07-14T18:06:42.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:41 smithi150 ceph-mon[128853]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-14T18:06:43.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:43 smithi115 ceph-mon[127451]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-14T18:06:43.677 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:43 smithi187 ceph-mon[129317]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-14T18:06:43.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:43 smithi150 ceph-mon[128853]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-14T18:06:46.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:45 smithi115 ceph-mon[127451]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-14T18:06:46.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:46.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:46.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:46.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:45 smithi187 ceph-mon[129317]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-14T18:06:46.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:46.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:46.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:46.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:45 smithi150 ceph-mon[128853]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-14T18:06:46.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:46.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:46.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:47.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.899+0000 7f75d4ff9700 1 -- 172.21.15.115:0/2365886057 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f75d8060a80 con 0x7f75c00a1140 2023-07-14T18:06:47.933 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 3 on host 'smithi115' 2023-07-14T18:06:47.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75c00a1140 msgr2=0x7f75c00a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:47.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75c00a1140 0x7f75c00a35f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00c010 tx=0x7f75cc005e80 comp rx=0 tx=0).stop 2023-07-14T18:06:47.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d8101020 msgr2=0x7f75d8195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d8101020 0x7f75d8195e60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f75c800d980 tx=0x7f75c800dc90 comp rx=0 tx=0).stop 2023-07-14T18:06:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 shutdown_connections 2023-07-14T18:06:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f75c00a1140 0x7f75c00a35f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d81019b0 0x7f75d819cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d80fdfa0 0x7f75d8195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 --2- 172.21.15.115:0/2365886057 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f75d8101020 0x7f75d8195e60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 >> 172.21.15.115:0/2365886057 conn(0x7f75d80f9990 msgr2=0x7f75d8104590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:06:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 shutdown_connections 2023-07-14T18:06:47.938 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:06:47.903+0000 7f75de0a6700 1 -- 172.21.15.115:0/2365886057 wait complete. 2023-07-14T18:06:48.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:47 smithi187 ceph-mon[129317]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T18:06:48.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:47 smithi115 ceph-mon[127451]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T18:06:48.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.278 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.279 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.280 DEBUG:teuthology.orchestra.run.smithi115:osd.3> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.3.service 2023-07-14T18:06:48.282 INFO:tasks.cephadm:Deploying osd.4 on smithi150 with /dev/vg_nvme/lv_3... 2023-07-14T18:06:48.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T18:06:48.311 INFO:journalctl@ceph.osd.3.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:06:48.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:47 smithi150 ceph-mon[128853]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T18:06:48.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.316 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.317 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:48.847 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:06:49.600 INFO:journalctl@ceph.osd.3.smithi115.stdout:Jul 14 18:06:49 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-3[145807]: 2023-07-14T18:06:49.216+0000 7f1330973640 -1 osd.3 0 log_to_monitors true 2023-07-14T18:06:50.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:49 smithi115 ceph-mon[127451]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-14T18:06:50.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:49 smithi115 ceph-mon[127451]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:06:50.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:49 smithi150 ceph-mon[128853]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-14T18:06:50.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:49 smithi150 ceph-mon[128853]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:06:50.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:49 smithi187 ceph-mon[129317]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-14T18:06:50.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:49 smithi187 ceph-mon[129317]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:06:50.214 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T18:06:50.214 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T18:06:50.215 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-07-14T18:06:50.215 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-07-14T18:06:50.215 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111479 s, 941 MB/s 2023-07-14T18:06:50.215 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-07-14T18:06:50.500 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2023-07-14T18:06:51.095 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:06:51.278 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T18:06:51.279 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T18:06:51.279 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:51.279 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:06:51.279 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.279 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.280 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:51.280 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:51.280 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.280 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:51.280 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:51.280 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.280 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:51.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:51.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:51.281 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.350 INFO:journalctl@ceph.osd.3.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-3[145807]: 2023-07-14T18:06:50.907+0000 7f1325feb700 -1 osd.3 0 waiting for initial osdmap 2023-07-14T18:06:51.350 INFO:journalctl@ceph.osd.3.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-3[145807]: 2023-07-14T18:06:50.913+0000 7f131ddd9700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:06:51.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T18:06:51.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T18:06:51.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:51.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:06:51.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:51.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:51.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:51.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:51.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:51.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:51.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:51.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T18:06:51.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T18:06:51.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:51.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:06:51.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:51.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:06:51.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:51.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:51.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:51.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:06:51.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:51.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:06:51.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:06:52.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.135+0000 7f108098b700 1 -- 172.21.15.150:0/973131197 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f107c0febb0 msgr2=0x7f107c0fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:52.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.135+0000 7f108098b700 1 --2- 172.21.15.150:0/973131197 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f107c0febb0 0x7f107c0fefc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1070009990 tx=0x7f1070009ca0 comp rx=0 tx=0).stop 2023-07-14T18:06:52.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.135+0000 7f108098b700 1 -- 172.21.15.150:0/973131197 shutdown_connections 2023-07-14T18:06:52.170 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.135+0000 7f108098b700 1 --2- 172.21.15.150:0/973131197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c100740 0x7f107c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.135+0000 7f108098b700 1 --2- 172.21.15.150:0/973131197 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f107c0febb0 0x7f107c0fefc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.135+0000 7f108098b700 1 --2- 172.21.15.150:0/973131197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107c0ffdb0 0x7f107c100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.135+0000 7f108098b700 1 -- 172.21.15.150:0/973131197 >> 172.21.15.150:0/973131197 conn(0x7f107c0fa1e0 msgr2=0x7f107c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:06:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.135+0000 7f108098b700 1 -- 172.21.15.150:0/973131197 shutdown_connections 2023-07-14T18:06:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.135+0000 7f108098b700 1 -- 172.21.15.150:0/973131197 wait complete. 2023-07-14T18:06:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.136+0000 7f108098b700 1 Processor -- start 2023-07-14T18:06:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.136+0000 7f108098b700 1 -- start start 2023-07-14T18:06:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.136+0000 7f108098b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f107c0febb0 0x7f107c1932b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.136+0000 7f108098b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0ffdb0 0x7f107c1937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.136+0000 7f108098b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107c100740 0x7f107c18fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.136+0000 7f108098b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c1905d0 con 0x7f107c0febb0 2023-07-14T18:06:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.136+0000 7f108098b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c190740 con 0x7f107c100740 2023-07-14T18:06:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.136+0000 7f108098b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c190900 con 0x7f107c0ffdb0 2023-07-14T18:06:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.136+0000 7f107ad9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107c100740 0x7f107c18fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f107ad9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107c100740 0x7f107c18fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53552/0 (socket says 172.21.15.150:53552) 2023-07-14T18:06:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f107ad9d700 1 -- 172.21.15.150:0/1914735145 learned_addr learned my addr 172.21.15.150:0/1914735145 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-07-14T18:06:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f1079d9b700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0ffdb0 0x7f107c1937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f107a59c700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f107c0febb0 0x7f107c1932b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f107ad9d700 1 -- 172.21.15.150:0/1914735145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0ffdb0 msgr2=0x7f107c1937f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f107ad9d700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0ffdb0 0x7f107c1937f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f107ad9d700 1 -- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f107c0febb0 msgr2=0x7f107c1932b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:06:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f107ad9d700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f107c0febb0 0x7f107c1932b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:06:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f107ad9d700 1 -- 172.21.15.150:0/1914735145 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c009520 con 0x7f107c100740 2023-07-14T18:06:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f107ad9d700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107c100740 0x7f107c18fe30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f107400ef30 tx=0x7f107400c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:06:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f106b7fe700 1 -- 172.21.15.150:0/1914735145 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074018070 con 0x7f107c100740 2023-07-14T18:06:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f106b7fe700 1 -- 172.21.15.150:0/1914735145 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f107400f040 con 0x7f107c100740 2023-07-14T18:06:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10700095f0 con 0x7f107c100740 2023-07-14T18:06:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f106b7fe700 1 -- 172.21.15.150:0/1914735145 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074003d00 con 0x7f107c100740 2023-07-14T18:06:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.137+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c1a42f0 con 0x7f107c100740 2023-07-14T18:06:52.175 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.138+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c10a220 con 0x7f107c100740 2023-07-14T18:06:52.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.139+0000 7f106b7fe700 1 -- 172.21.15.150:0/1914735145 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f107400caa0 con 0x7f107c100740 2023-07-14T18:06:52.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.140+0000 7f106b7fe700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a1150 0x7f10640a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:06:52.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.140+0000 7f106b7fe700 1 -- 172.21.15.150:0/1914735145 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3702+0+0 (secure 0 0 0) 0x7f10740cb510 con 0x7f107c100740 2023-07-14T18:06:52.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.140+0000 7f107a59c700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a1150 0x7f10640a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:06:52.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.140+0000 7f107a59c700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a1150 0x7f10640a3600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1070005910 tx=0x7f107000b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:06:52.176 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.144+0000 7f106b7fe700 1 -- 172.21.15.150:0/1914735145 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1074095e50 con 0x7f107c100740 2023-07-14T18:06:52.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:51 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi115 2023-07-14T18:06:52.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:51 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi115 to 8584M 2023-07-14T18:06:52.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:51 smithi115 ceph-mon[127451]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:52.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:51 smithi115 ceph-mon[127451]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:06:52.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:51 smithi115 ceph-mon[127451]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T18:06:52.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:52.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:52.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:51 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi115 2023-07-14T18:06:52.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:51 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi115 to 8584M 2023-07-14T18:06:52.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:51 smithi150 ceph-mon[128853]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:52.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:51 smithi150 ceph-mon[128853]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:06:52.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:51 smithi150 ceph-mon[128853]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T18:06:52.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:52.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:52.369 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:06:52.337+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f107c1a4930 con 0x7f10640a1150 2023-07-14T18:06:52.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:51 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi115 2023-07-14T18:06:52.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:51 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi115 to 8584M 2023-07-14T18:06:52.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:51 smithi187 ceph-mon[129317]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:06:52.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:51 smithi187 ceph-mon[129317]: from='osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:06:52.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:51 smithi187 ceph-mon[129317]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T18:06:52.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:52.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:53.202 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:52 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:06:53.202 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:52 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:06:53.202 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:52 smithi150 ceph-mon[128853]: osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] boot 2023-07-14T18:06:53.202 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:52 smithi150 ceph-mon[128853]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T18:06:53.203 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:53.203 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:06:53.203 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:06:53.203 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:53.203 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:52 smithi150 ceph-mon[128853]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T18:06:53.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:52 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:06:53.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:52 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:06:53.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:52 smithi115 ceph-mon[127451]: osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] boot 2023-07-14T18:06:53.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:52 smithi115 ceph-mon[127451]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T18:06:53.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:53.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:06:53.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:06:53.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:53.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:52 smithi115 ceph-mon[127451]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T18:06:53.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:52 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:06:53.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:52 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:06:53.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:52 smithi187 ceph-mon[129317]: osd.3 [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] boot 2023-07-14T18:06:53.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:52 smithi187 ceph-mon[129317]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T18:06:53.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:06:53.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:06:53.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:06:53.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:06:53.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:52 smithi187 ceph-mon[129317]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T18:06:54.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:53 smithi150 ceph-mon[128853]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:06:54.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:53 smithi150 ceph-mon[128853]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:06:54.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:53 smithi150 ceph-mon[128853]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T18:06:54.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:53 smithi115 ceph-mon[127451]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:06:54.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:53 smithi115 ceph-mon[127451]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:06:54.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:53 smithi115 ceph-mon[127451]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T18:06:54.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:53 smithi187 ceph-mon[129317]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:06:54.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:53 smithi187 ceph-mon[129317]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:06:54.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:53 smithi187 ceph-mon[129317]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T18:06:55.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:55 smithi187 ceph-mon[129317]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T18:06:55.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:55 smithi187 ceph-mon[129317]: from='client.? 172.21.15.150:0/599542471' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82386140-1f12-43e9-9d70-6ca597009e77"}]: dispatch 2023-07-14T18:06:55.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:55 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82386140-1f12-43e9-9d70-6ca597009e77"}]: dispatch 2023-07-14T18:06:55.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:55 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82386140-1f12-43e9-9d70-6ca597009e77"}]': finished 2023-07-14T18:06:55.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:55 smithi187 ceph-mon[129317]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T18:06:55.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:06:55.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:55 smithi187 ceph-mon[129317]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:06:55.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:55 smithi187 ceph-mon[129317]: from='client.? 172.21.15.150:0/2602888657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:06:55.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:55 smithi115 ceph-mon[127451]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T18:06:55.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:55 smithi115 ceph-mon[127451]: from='client.? 172.21.15.150:0/599542471' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82386140-1f12-43e9-9d70-6ca597009e77"}]: dispatch 2023-07-14T18:06:55.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:55 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82386140-1f12-43e9-9d70-6ca597009e77"}]: dispatch 2023-07-14T18:06:55.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:55 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82386140-1f12-43e9-9d70-6ca597009e77"}]': finished 2023-07-14T18:06:55.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:55 smithi115 ceph-mon[127451]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T18:06:55.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:06:55.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:55 smithi115 ceph-mon[127451]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:06:55.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:55 smithi115 ceph-mon[127451]: from='client.? 172.21.15.150:0/2602888657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:06:55.865 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:55 smithi150 ceph-mon[128853]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T18:06:55.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:55 smithi150 ceph-mon[128853]: from='client.? 172.21.15.150:0/599542471' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82386140-1f12-43e9-9d70-6ca597009e77"}]: dispatch 2023-07-14T18:06:55.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:55 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82386140-1f12-43e9-9d70-6ca597009e77"}]: dispatch 2023-07-14T18:06:55.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:55 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82386140-1f12-43e9-9d70-6ca597009e77"}]': finished 2023-07-14T18:06:55.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:55 smithi150 ceph-mon[128853]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T18:06:55.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:06:55.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:55 smithi150 ceph-mon[128853]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:06:55.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:55 smithi150 ceph-mon[128853]: from='client.? 172.21.15.150:0/2602888657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:06:58.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:57 smithi187 ceph-mon[129317]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-14T18:06:58.212 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:57 smithi150 ceph-mon[128853]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-14T18:06:58.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:57 smithi115 ceph-mon[127451]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-14T18:07:00.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:06:59 smithi115 ceph-mon[127451]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:07:00.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:06:59 smithi150 ceph-mon[128853]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:07:00.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:06:59 smithi187 ceph-mon[129317]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:07:01.115 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:00 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T18:07:01.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:00 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:01.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:00 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T18:07:01.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:00 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:01.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:00 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T18:07:01.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:00 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:02.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:01 smithi115 ceph-mon[127451]: Deploying daemon osd.4 on smithi150 2023-07-14T18:07:02.253 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:01 smithi115 ceph-mon[127451]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:07:02.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:01 smithi150 ceph-mon[128853]: Deploying daemon osd.4 on smithi150 2023-07-14T18:07:02.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:01 smithi150 ceph-mon[128853]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:07:02.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:01 smithi187 ceph-mon[129317]: Deploying daemon osd.4 on smithi150 2023-07-14T18:07:02.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:01 smithi187 ceph-mon[129317]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:07:03.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:03 smithi187 ceph-mon[129317]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-14T18:07:03.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:03 smithi115 ceph-mon[127451]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-14T18:07:03.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:03 smithi150 ceph-mon[128853]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-14T18:07:05.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:05.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:05.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:05.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:05 smithi150 ceph-mon[128853]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-14T18:07:06.100 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:06.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:06.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:06.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:05 smithi115 ceph-mon[127451]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-14T18:07:06.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:06.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:06.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:06.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:05 smithi187 ceph-mon[129317]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-14T18:07:07.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.343+0000 7f106b7fe700 1 -- 172.21.15.150:0/1914735145 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f107c1a4930 con 0x7f10640a1150 2023-07-14T18:07:07.377 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 4 on host 'smithi150' 2023-07-14T18:07:07.378 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a1150 msgr2=0x7f10640a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:07.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a1150 0x7f10640a3600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1070005910 tx=0x7f107000b260 comp rx=0 tx=0).stop 2023-07-14T18:07:07.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107c100740 msgr2=0x7f107c18fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:07.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107c100740 0x7f107c18fe30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f107400ef30 tx=0x7f107400c450 comp rx=0 tx=0).stop 2023-07-14T18:07:07.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 shutdown_connections 2023-07-14T18:07:07.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a1150 0x7f10640a3600 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:07.379 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f107c100740 0x7f107c18fe30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:07.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f107c0febb0 0x7f107c1932b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:07.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 --2- 172.21.15.150:0/1914735145 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f107c0ffdb0 0x7f107c1937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:07.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.345+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 >> 172.21.15.150:0/1914735145 conn(0x7f107c0fa1e0 msgr2=0x7f107c068120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:07:07.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.346+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 shutdown_connections 2023-07-14T18:07:07.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:07.346+0000 7f108098b700 1 -- 172.21.15.150:0/1914735145 wait complete. 2023-07-14T18:07:07.787 DEBUG:teuthology.orchestra.run.smithi150:osd.4> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.4.service 2023-07-14T18:07:07.789 INFO:tasks.cephadm:Deploying osd.5 on smithi187 with /dev/vg_nvme/lv_3... 2023-07-14T18:07:07.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T18:07:07.819 INFO:journalctl@ceph.osd.4.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:07:08.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:07 smithi187 ceph-mon[129317]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T18:07:08.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:07 smithi115 ceph-mon[127451]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T18:07:08.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.351 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:07:08.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:07 smithi150 ceph-mon[128853]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T18:07:08.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.352 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:08.980 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:07:08 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-4[139346]: 2023-07-14T18:07:08.756+0000 7f2f29ea8640 -1 osd.4 0 log_to_monitors true 2023-07-14T18:07:09.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:08 smithi187 ceph-mon[129317]: from='osd.4 [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:07:09.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:08 smithi187 ceph-mon[129317]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:07:09.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:08 smithi115 ceph-mon[127451]: from='osd.4 [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:07:09.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:08 smithi115 ceph-mon[127451]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:07:09.365 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:08 smithi150 ceph-mon[128853]: from='osd.4 [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:07:09.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:08 smithi150 ceph-mon[128853]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:07:09.837 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T18:07:09.837 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T18:07:09.837 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2023-07-14T18:07:09.837 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2023-07-14T18:07:09.837 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109518 s, 957 MB/s 2023-07-14T18:07:09.838 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2023-07-14T18:07:10.178 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:07:10.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T18:07:10.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T18:07:10.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:10.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: from='osd.4 [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:07:10.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:07:10.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:10.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:10.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:10.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:10.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi187:vg_nvme/lv_3 2023-07-14T18:07:10.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:07:10.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T18:07:10.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T18:07:10.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:10.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: from='osd.4 [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:07:10.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:07:10.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:10.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:10.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:10.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:10.366 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-4[139346]: 2023-07-14T18:07:09.963+0000 7f2f1f520700 -1 osd.4 0 waiting for initial osdmap 2023-07-14T18:07:10.366 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-4[139346]: 2023-07-14T18:07:09.970+0000 7f2f19312700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:07:10.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:07:10.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T18:07:10.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T18:07:10.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:10.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: from='osd.4 [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:07:10.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:07:10.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:10.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:10.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:10.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:10.856 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:07:11.343 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi150 2023-07-14T18:07:11.343 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi150 to 8584M 2023-07-14T18:07:11.343 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:07:11.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T18:07:11.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:11.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:11.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:11.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:11.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:11.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:11.345 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:11.345 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:11.345 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:11.345 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:11.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi150 2023-07-14T18:07:11.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi150 to 8584M 2023-07-14T18:07:11.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:07:11.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T18:07:11.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:11.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:11.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:11.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:11.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi150 2023-07-14T18:07:11.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi150 to 8584M 2023-07-14T18:07:11.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:07:11.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T18:07:11.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:11.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:11.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:11.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:11.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:11.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:11.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:11.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:11.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:11.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:12.036 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.002+0000 7f6a82e99700 1 -- 172.21.15.187:0/4290588170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 msgr2=0x7f6a7c100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:12.037 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.002+0000 7f6a82e99700 1 --2- 172.21.15.187:0/4290588170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 0x7f6a7c100200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009990 tx=0x7f6a68009ca0 comp rx=0 tx=0).stop 2023-07-14T18:07:12.037 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.002+0000 7f6a82e99700 1 -- 172.21.15.187:0/4290588170 shutdown_connections 2023-07-14T18:07:12.037 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.002+0000 7f6a82e99700 1 --2- 172.21.15.187:0/4290588170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a7c100740 0x7f6a7c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:12.037 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.002+0000 7f6a82e99700 1 --2- 172.21.15.187:0/4290588170 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a7c0febb0 0x7f6a7c0fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:12.037 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.002+0000 7f6a82e99700 1 --2- 172.21.15.187:0/4290588170 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 0x7f6a7c100200 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:12.037 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.002+0000 7f6a82e99700 1 -- 172.21.15.187:0/4290588170 >> 172.21.15.187:0/4290588170 conn(0x7f6a7c0fa1e0 msgr2=0x7f6a7c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:07:12.038 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.002+0000 7f6a82e99700 1 -- 172.21.15.187:0/4290588170 shutdown_connections 2023-07-14T18:07:12.038 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.002+0000 7f6a82e99700 1 -- 172.21.15.187:0/4290588170 wait complete. 2023-07-14T18:07:12.038 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a82e99700 1 Processor -- start 2023-07-14T18:07:12.040 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a82e99700 1 -- start start 2023-07-14T18:07:12.040 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a82e99700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a7c0febb0 0x7f6a7c0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:12.040 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a82e99700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 0x7f6a7c19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:12.040 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a82e99700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a7c100740 0x7f6a7c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:12.040 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a82e99700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c19d3d0 con 0x7f6a7c100740 2023-07-14T18:07:12.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a82e99700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c19d540 con 0x7f6a7c0febb0 2023-07-14T18:07:12.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a82e99700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c19d700 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a7bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 0x7f6a7c19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:12.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a7bfff700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 0x7f6a7c19a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:58552/0 (socket says 172.21.15.187:58552) 2023-07-14T18:07:12.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a7bfff700 1 -- 172.21.15.187:0/713462858 learned_addr learned my addr 172.21.15.187:0/713462858 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:07:12.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a80c35700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a7c0febb0 0x7f6a7c0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:12.041 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a81436700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a7c100740 0x7f6a7c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:12.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a7bfff700 1 -- 172.21.15.187:0/713462858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a7c0febb0 msgr2=0x7f6a7c0694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:12.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a7bfff700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a7c0febb0 0x7f6a7c0694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:12.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a7bfff700 1 -- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a7c100740 msgr2=0x7f6a7c19cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:12.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a7bfff700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a7c100740 0x7f6a7c19cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:12.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a7bfff700 1 -- 172.21.15.187:0/713462858 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680095f0 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a81436700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a7c100740 0x7f6a7c19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:07:12.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.003+0000 7f6a7bfff700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 0x7f6a7c19a5f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6a68006010 tx=0x7f6a68003a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:07:12.042 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.004+0000 7f6a79ffb700 1 -- 172.21.15.187:0/713462858 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6801d070 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.043 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.004+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7c19d980 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.043 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.004+0000 7f6a79ffb700 1 -- 172.21.15.187:0/713462858 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6a6800bd70 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.043 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.004+0000 7f6a79ffb700 1 -- 172.21.15.187:0/713462858 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68022cb0 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.043 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.004+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7c19dd90 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.043 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.005+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7c1061f0 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.043 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.006+0000 7f6a79ffb700 1 -- 172.21.15.187:0/713462858 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f6a68022470 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.043 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.006+0000 7f6a79ffb700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6a6c0a1150 0x7f6a6c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:12.044 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.006+0000 7f6a79ffb700 1 -- 172.21.15.187:0/713462858 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f6a680cced0 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.044 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.006+0000 7f6a80c35700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6a6c0a1150 0x7f6a6c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:12.044 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.007+0000 7f6a80c35700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6a6c0a1150 0x7f6a6c0a3600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0ffc10 tx=0x7f6a7000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:07:12.045 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.011+0000 7f6a79ffb700 1 -- 172.21.15.187:0/713462858 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6a68097720 con 0x7f6a7c0ffdb0 2023-07-14T18:07:12.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:11 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:07:12.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:11 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:07:12.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:11 smithi187 ceph-mon[129317]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:07:12.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:12.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:11 smithi187 ceph-mon[129317]: osd.4 [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] boot 2023-07-14T18:07:12.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:11 smithi187 ceph-mon[129317]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T18:07:12.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:12.236 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:12.202+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a7c060a80 con 0x7f6a6c0a1150 2023-07-14T18:07:12.350 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:11 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:07:12.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:11 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:07:12.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:11 smithi115 ceph-mon[127451]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:07:12.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:12.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:11 smithi115 ceph-mon[127451]: osd.4 [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] boot 2023-07-14T18:07:12.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:11 smithi115 ceph-mon[127451]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T18:07:12.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:12.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:11 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:07:12.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:11 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:07:12.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:11 smithi150 ceph-mon[128853]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:07:12.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:12.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:11 smithi150 ceph-mon[128853]: osd.4 [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] boot 2023-07-14T18:07:12.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:11 smithi150 ceph-mon[128853]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T18:07:12.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:07:13.326 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:13 smithi187 ceph-mon[129317]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T18:07:13.327 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:13 smithi187 ceph-mon[129317]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:07:13.327 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:13 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:07:13.327 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:13 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:07:13.328 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:13 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:13.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:13 smithi115 ceph-mon[127451]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T18:07:13.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:13 smithi115 ceph-mon[127451]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:07:13.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:13 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:07:13.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:13 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:07:13.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:13 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:13.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:13 smithi150 ceph-mon[128853]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T18:07:13.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:13 smithi150 ceph-mon[128853]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:07:13.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:13 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:07:13.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:13 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:07:13.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:13 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:14.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:14 smithi115 ceph-mon[127451]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:14.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:14 smithi150 ceph-mon[128853]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:14.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:14 smithi187 ceph-mon[129317]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:15.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:15 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/3026053670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9af1e805-b05f-422e-8f27-f95d74e9fec6"}]: dispatch 2023-07-14T18:07:15.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:15 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9af1e805-b05f-422e-8f27-f95d74e9fec6"}]: dispatch 2023-07-14T18:07:15.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:15 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9af1e805-b05f-422e-8f27-f95d74e9fec6"}]': finished 2023-07-14T18:07:15.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:15 smithi115 ceph-mon[127451]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T18:07:15.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:15.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:15 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/3026053670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9af1e805-b05f-422e-8f27-f95d74e9fec6"}]: dispatch 2023-07-14T18:07:15.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:15 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9af1e805-b05f-422e-8f27-f95d74e9fec6"}]: dispatch 2023-07-14T18:07:15.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:15 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9af1e805-b05f-422e-8f27-f95d74e9fec6"}]': finished 2023-07-14T18:07:15.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:15 smithi150 ceph-mon[128853]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T18:07:15.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:15.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:15 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/3026053670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9af1e805-b05f-422e-8f27-f95d74e9fec6"}]: dispatch 2023-07-14T18:07:15.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:15 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9af1e805-b05f-422e-8f27-f95d74e9fec6"}]: dispatch 2023-07-14T18:07:15.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:15 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9af1e805-b05f-422e-8f27-f95d74e9fec6"}]': finished 2023-07-14T18:07:15.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:15 smithi187 ceph-mon[129317]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T18:07:15.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:16.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:16 smithi115 ceph-mon[127451]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:16.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:16 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/3160010897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:07:16.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:16 smithi150 ceph-mon[128853]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:16.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:16 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/3160010897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:07:16.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:16 smithi187 ceph-mon[129317]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:16.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:16 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/3160010897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:07:18.600 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:18 smithi115 ceph-mon[127451]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:18.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:18 smithi150 ceph-mon[128853]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:18.640 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:18 smithi187 ceph-mon[129317]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:20.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:20 smithi187 ceph-mon[129317]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:20.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:20 smithi115 ceph-mon[127451]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:20.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:20 smithi150 ceph-mon[128853]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:21.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:21 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T18:07:21.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:21 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:21.614 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:21 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T18:07:21.614 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:21 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:21.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:21 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T18:07:21.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:21 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:22.490 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:22 smithi187 ceph-mon[129317]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:22.490 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:22 smithi187 ceph-mon[129317]: Deploying daemon osd.5 on smithi187 2023-07-14T18:07:22.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:22 smithi115 ceph-mon[127451]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:22.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:22 smithi115 ceph-mon[127451]: Deploying daemon osd.5 on smithi187 2023-07-14T18:07:22.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:22 smithi150 ceph-mon[128853]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:22.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:22 smithi150 ceph-mon[128853]: Deploying daemon osd.5 on smithi187 2023-07-14T18:07:23.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:23 smithi187 ceph-mon[129317]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:23.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:23 smithi115 ceph-mon[127451]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:23.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:23 smithi150 ceph-mon[128853]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:26.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:26 smithi187 ceph-mon[129317]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:26.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:26 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:26.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:26 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:26.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:26 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:26.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:26 smithi115 ceph-mon[127451]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:26.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:26 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:26.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:26 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:26.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:26 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:26.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:26 smithi150 ceph-mon[128853]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:26.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:26 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:26.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:26 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:26.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:26 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.043 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 5 on host 'smithi187' 2023-07-14T18:07:28.044 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.009+0000 7f6a79ffb700 1 -- 172.21.15.187:0/713462858 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6a7c060a80 con 0x7f6a6c0a1150 2023-07-14T18:07:28.046 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6a6c0a1150 msgr2=0x7f6a6c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:28.046 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6a6c0a1150 0x7f6a6c0a3600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c0ffc10 tx=0x7f6a7000b410 comp rx=0 tx=0).stop 2023-07-14T18:07:28.046 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 msgr2=0x7f6a7c19a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:28.046 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 0x7f6a7c19a5f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6a68006010 tx=0x7f6a68003a50 comp rx=0 tx=0).stop 2023-07-14T18:07:28.046 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 shutdown_connections 2023-07-14T18:07:28.046 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6a6c0a1150 0x7f6a6c0a3600 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:28.046 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6a7c100740 0x7f6a7c19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:28.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a7c0febb0 0x7f6a7c0694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:28.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 --2- 172.21.15.187:0/713462858 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6a7c0ffdb0 0x7f6a7c19a5f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:28.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 >> 172.21.15.187:0/713462858 conn(0x7f6a7c0fa1e0 msgr2=0x7f6a7c103330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:07:28.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 shutdown_connections 2023-07-14T18:07:28.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:07:28.012+0000 7f6a82e99700 1 -- 172.21.15.187:0/713462858 wait complete. 2023-07-14T18:07:28.542 DEBUG:teuthology.orchestra.run.smithi187:osd.5> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.5.service 2023-07-14T18:07:28.544 INFO:tasks.cephadm:Deploying osd.6 on smithi115 with /dev/vg_nvme/lv_2... 2023-07-14T18:07:28.545 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T18:07:28.571 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:28 smithi187 ceph-mon[129317]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:28.572 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.572 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.572 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:28.572 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:28.572 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.572 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.573 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.573 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:28.577 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:28 smithi115 ceph-mon[127451]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:28.578 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.578 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.578 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:28.578 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:28.578 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.578 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.578 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.578 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:28.580 INFO:journalctl@ceph.osd.5.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:07:28.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:28 smithi150 ceph-mon[128853]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:28.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:28.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:28.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:28.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:29.115 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:07:29.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:29 smithi115 ceph-mon[127451]: from='osd.5 [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:07:29.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:29 smithi115 ceph-mon[127451]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:07:29.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:29 smithi150 ceph-mon[128853]: from='osd.5 [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:07:29.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:29 smithi150 ceph-mon[128853]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:07:29.672 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:29 smithi187 ceph-mon[129317]: from='osd.5 [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:07:29.673 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:29 smithi187 ceph-mon[129317]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:07:29.673 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:07:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:07:29.251+0000 7f8667453640 -1 osd.5 0 log_to_monitors true 2023-07-14T18:07:30.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:30 smithi115 ceph-mon[127451]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:30.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:30 smithi115 ceph-mon[127451]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T18:07:30.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:30 smithi115 ceph-mon[127451]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T18:07:30.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:30.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:30 smithi115 ceph-mon[127451]: from='osd.5 [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:07:30.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:30 smithi115 ceph-mon[127451]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:07:30.562 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:07:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:07:30.294+0000 7f865cacb700 -1 osd.5 0 waiting for initial osdmap 2023-07-14T18:07:30.562 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:07:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:07:30.299+0000 7f86548b9700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:07:30.563 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:30 smithi187 ceph-mon[129317]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:30.563 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:30 smithi187 ceph-mon[129317]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T18:07:30.563 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:30 smithi187 ceph-mon[129317]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T18:07:30.563 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:30.563 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:30 smithi187 ceph-mon[129317]: from='osd.5 [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:07:30.563 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:30 smithi187 ceph-mon[129317]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:07:30.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:30 smithi150 ceph-mon[128853]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:30.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:30 smithi150 ceph-mon[128853]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T18:07:30.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:30 smithi150 ceph-mon[128853]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T18:07:30.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:30.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:30 smithi150 ceph-mon[128853]: from='osd.5 [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:07:30.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:30 smithi150 ceph-mon[128853]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:07:30.794 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T18:07:30.795 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T18:07:30.795 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2023-07-14T18:07:30.795 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2023-07-14T18:07:30.795 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111053 s, 944 MB/s 2023-07-14T18:07:30.795 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2023-07-14T18:07:31.276 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_2 2023-07-14T18:07:31.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:07:31.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T18:07:31.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:31.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:31.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:31.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:31.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:31.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:31.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:31 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:07:31.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T18:07:31.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:31.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:31.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:31.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:31.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:31.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:31.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:31 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:07:31.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T18:07:31.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:31.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:31.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:31.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:31.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:31.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:31.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:31 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:31.871 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:07:32.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:32 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:07:32.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:32 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:07:32.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:32 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi187 2023-07-14T18:07:32.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:32 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi187 to 8584M 2023-07-14T18:07:32.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:32 smithi115 ceph-mon[127451]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:32.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:32 smithi115 ceph-mon[127451]: osd.5 [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] boot 2023-07-14T18:07:32.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:32 smithi115 ceph-mon[127451]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T18:07:32.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:32.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:32 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:07:32.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:32 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:07:32.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:32 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi187 2023-07-14T18:07:32.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:32 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi187 to 8584M 2023-07-14T18:07:32.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:32 smithi150 ceph-mon[128853]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:32.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:32 smithi150 ceph-mon[128853]: osd.5 [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] boot 2023-07-14T18:07:32.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:32 smithi150 ceph-mon[128853]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T18:07:32.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:32.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:32 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:07:32.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:32 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:07:32.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:32 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi187 2023-07-14T18:07:32.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:32 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi187 to 8584M 2023-07-14T18:07:32.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:32 smithi187 ceph-mon[129317]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:07:32.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:32 smithi187 ceph-mon[129317]: osd.5 [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] boot 2023-07-14T18:07:32.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:32 smithi187 ceph-mon[129317]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T18:07:32.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:07:32.942 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.908+0000 7fc5be23f700 1 -- 172.21.15.115:0/1553353929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b8100740 msgr2=0x7fc5b8107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:32.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.908+0000 7fc5be23f700 1 --2- 172.21.15.115:0/1553353929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b8100740 0x7fc5b8107000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8009990 tx=0x7fc5a8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:07:32.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.908+0000 7fc5be23f700 1 -- 172.21.15.115:0/1553353929 shutdown_connections 2023-07-14T18:07:32.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.908+0000 7fc5be23f700 1 --2- 172.21.15.115:0/1553353929 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b8100740 0x7fc5b8107000 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:32.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.908+0000 7fc5be23f700 1 --2- 172.21.15.115:0/1553353929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b80febb0 0x7fc5b80fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:32.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.908+0000 7fc5be23f700 1 --2- 172.21.15.115:0/1553353929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5b80ffdb0 0x7fc5b8100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:32.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.908+0000 7fc5be23f700 1 -- 172.21.15.115:0/1553353929 >> 172.21.15.115:0/1553353929 conn(0x7fc5b80fa1e0 msgr2=0x7fc5b80fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:07:32.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.908+0000 7fc5be23f700 1 -- 172.21.15.115:0/1553353929 shutdown_connections 2023-07-14T18:07:32.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.908+0000 7fc5be23f700 1 -- 172.21.15.115:0/1553353929 wait complete. 2023-07-14T18:07:32.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5be23f700 1 Processor -- start 2023-07-14T18:07:32.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5be23f700 1 -- start start 2023-07-14T18:07:32.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5be23f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b80febb0 0x7fc5b8195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:32.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5be23f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5b80ffdb0 0x7fc5b8195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:32.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5be23f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b8100740 0x7fc5b819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:32.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5be23f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b819d5b0 con 0x7fc5b80febb0 2023-07-14T18:07:32.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5be23f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b819d6f0 con 0x7fc5b8100740 2023-07-14T18:07:32.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5be23f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b819d880 con 0x7fc5b80ffdb0 2023-07-14T18:07:32.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b80febb0 0x7fc5b8195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:32.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b80febb0 0x7fc5b8195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51852/0 (socket says 172.21.15.115:51852) 2023-07-14T18:07:32.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5b77fe700 1 -- 172.21.15.115:0/535497492 learned_addr learned my addr 172.21.15.115:0/535497492 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:07:32.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5b7fff700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b8100740 0x7fc5b819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:32.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5b6ffd700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5b80ffdb0 0x7fc5b8195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:32.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.909+0000 7fc5b77fe700 1 -- 172.21.15.115:0/535497492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5b80ffdb0 msgr2=0x7fc5b8195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:32.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5b77fe700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5b80ffdb0 0x7fc5b8195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:32.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5b77fe700 1 -- 172.21.15.115:0/535497492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b8100740 msgr2=0x7fc5b819cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:32.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5b77fe700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b8100740 0x7fc5b819cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:32.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5b77fe700 1 -- 172.21.15.115:0/535497492 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a80095f0 con 0x7fc5b80febb0 2023-07-14T18:07:32.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5b7fff700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b8100740 0x7fc5b819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:07:32.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5b77fe700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b80febb0 0x7fc5b8195a20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000b9f0 tx=0x7fc5b000bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:07:32.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5b4ff9700 1 -- 172.21.15.115:0/535497492 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000ca30 con 0x7fc5b80febb0 2023-07-14T18:07:32.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b819db60 con 0x7fc5b80febb0 2023-07-14T18:07:32.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5b4ff9700 1 -- 172.21.15.115:0/535497492 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc5b000cb90 con 0x7fc5b80febb0 2023-07-14T18:07:32.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5b4ff9700 1 -- 172.21.15.115:0/535497492 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00125a0 con 0x7fc5b80febb0 2023-07-14T18:07:32.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.910+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b819e060 con 0x7fc5b80febb0 2023-07-14T18:07:32.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.911+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b8106260 con 0x7fc5b80febb0 2023-07-14T18:07:32.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.912+0000 7fc5b4ff9700 1 -- 172.21.15.115:0/535497492 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fc5b000cd00 con 0x7fc5b80febb0 2023-07-14T18:07:32.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.913+0000 7fc5b4ff9700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5a00a1150 0x7fc5a00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:32.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.913+0000 7fc5b6ffd700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5a00a1150 0x7fc5a00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:32.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.913+0000 7fc5b4ff9700 1 -- 172.21.15.115:0/535497492 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fc5b0006e40 con 0x7fc5b80febb0 2023-07-14T18:07:32.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.913+0000 7fc5b6ffd700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5a00a1150 0x7fc5a00a3600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac006fd0 tx=0x7fc5ac008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:07:32.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:32.916+0000 7fc5b4ff9700 1 -- 172.21.15.115:0/535497492 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc5b0096230 con 0x7fc5b80febb0 2023-07-14T18:07:33.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:33.109+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5b8060a80 con 0x7fc5a00a1150 2023-07-14T18:07:33.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:33 smithi115 ceph-mon[127451]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T18:07:33.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:33 smithi115 ceph-mon[127451]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:33.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:33 smithi115 ceph-mon[127451]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:07:33.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:07:33.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:07:33.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:33.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:33 smithi150 ceph-mon[128853]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T18:07:33.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:33 smithi150 ceph-mon[128853]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:33.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:33 smithi150 ceph-mon[128853]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:07:33.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:07:33.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:07:33.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:33.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:33 smithi187 ceph-mon[129317]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T18:07:33.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:33 smithi187 ceph-mon[129317]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:33.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:33 smithi187 ceph-mon[129317]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:07:33.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:07:33.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:07:33.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:34.601 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:34 smithi115 ceph-mon[127451]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T18:07:34.616 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:34 smithi150 ceph-mon[128853]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T18:07:34.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:34 smithi187 ceph-mon[129317]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T18:07:35.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:35 smithi187 ceph-mon[129317]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T18:07:35.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:35 smithi187 ceph-mon[129317]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:35.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:35 smithi115 ceph-mon[127451]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T18:07:35.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:35 smithi115 ceph-mon[127451]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:35.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:35 smithi150 ceph-mon[128853]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T18:07:35.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:35 smithi150 ceph-mon[128853]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:36.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:36 smithi187 ceph-mon[129317]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T18:07:36.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:36 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/799343742' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bceb601-e8f0-440f-b674-3ce286cc4d3d"}]: dispatch 2023-07-14T18:07:36.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:36 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/799343742' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bceb601-e8f0-440f-b674-3ce286cc4d3d"}]': finished 2023-07-14T18:07:36.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:36 smithi187 ceph-mon[129317]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T18:07:36.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:36 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:36.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:36 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/4250716007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:07:36.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:36 smithi115 ceph-mon[127451]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T18:07:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:36 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/799343742' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bceb601-e8f0-440f-b674-3ce286cc4d3d"}]: dispatch 2023-07-14T18:07:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:36 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/799343742' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bceb601-e8f0-440f-b674-3ce286cc4d3d"}]': finished 2023-07-14T18:07:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:36 smithi115 ceph-mon[127451]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T18:07:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:36 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:36.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:36 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4250716007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:07:36.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:36 smithi150 ceph-mon[128853]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T18:07:36.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:36 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/799343742' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5bceb601-e8f0-440f-b674-3ce286cc4d3d"}]: dispatch 2023-07-14T18:07:36.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:36 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/799343742' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5bceb601-e8f0-440f-b674-3ce286cc4d3d"}]': finished 2023-07-14T18:07:36.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:36 smithi150 ceph-mon[128853]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T18:07:36.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:36 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:36.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:36 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/4250716007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:07:37.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:37 smithi187 ceph-mon[129317]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T18:07:37.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:37 smithi115 ceph-mon[127451]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T18:07:37.866 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:37 smithi150 ceph-mon[128853]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T18:07:40.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:39 smithi115 ceph-mon[127451]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2023-07-14T18:07:40.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:39 smithi150 ceph-mon[128853]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2023-07-14T18:07:40.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:39 smithi187 ceph-mon[129317]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2023-07-14T18:07:42.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:41 smithi115 ceph-mon[127451]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-14T18:07:42.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:41 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T18:07:42.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:41 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:42.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:41 smithi150 ceph-mon[128853]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-14T18:07:42.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:41 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T18:07:42.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:41 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:42.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:41 smithi187 ceph-mon[129317]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-14T18:07:42.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:41 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T18:07:42.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:41 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:43.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:42 smithi115 ceph-mon[127451]: Deploying daemon osd.6 on smithi115 2023-07-14T18:07:43.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:42 smithi150 ceph-mon[128853]: Deploying daemon osd.6 on smithi115 2023-07-14T18:07:43.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:42 smithi187 ceph-mon[129317]: Deploying daemon osd.6 on smithi115 2023-07-14T18:07:44.028 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:43 smithi115 ceph-mon[127451]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T18:07:44.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:43 smithi150 ceph-mon[128853]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T18:07:44.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:43 smithi187 ceph-mon[129317]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T18:07:46.214 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:45 smithi115 ceph-mon[127451]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-14T18:07:46.366 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:45 smithi150 ceph-mon[128853]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-14T18:07:46.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:45 smithi187 ceph-mon[129317]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-14T18:07:47.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:47.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:47.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:47.506 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:47 smithi115 ceph-mon[127451]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-14T18:07:47.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:47.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:47.679 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:47.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:47 smithi187 ceph-mon[129317]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-14T18:07:47.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:47.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:47.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:47.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:47 smithi150 ceph-mon[128853]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-14T18:07:49.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.412+0000 7fc5b4ff9700 1 -- 172.21.15.115:0/535497492 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc5b8060a80 con 0x7fc5a00a1150 2023-07-14T18:07:49.447 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 6 on host 'smithi115' 2023-07-14T18:07:49.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5a00a1150 msgr2=0x7fc5a00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:49.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5a00a1150 0x7fc5a00a3600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac006fd0 tx=0x7fc5ac008040 comp rx=0 tx=0).stop 2023-07-14T18:07:49.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b80febb0 msgr2=0x7fc5b8195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:49.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b80febb0 0x7fc5b8195a20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000b9f0 tx=0x7fc5b000bdb0 comp rx=0 tx=0).stop 2023-07-14T18:07:49.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 shutdown_connections 2023-07-14T18:07:49.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5a00a1150 0x7fc5a00a3600 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:49.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b8100740 0x7fc5b819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:49.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5b80febb0 0x7fc5b8195a20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:49.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 --2- 172.21.15.115:0/535497492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5b80ffdb0 0x7fc5b8195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:49.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 >> 172.21.15.115:0/535497492 conn(0x7fc5b80fa1e0 msgr2=0x7fc5b81033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:07:49.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 shutdown_connections 2023-07-14T18:07:49.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:07:49.415+0000 7fc5be23f700 1 -- 172.21.15.115:0/535497492 wait complete. 2023-07-14T18:07:49.856 DEBUG:teuthology.orchestra.run.smithi115:osd.6> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.6.service 2023-07-14T18:07:49.859 INFO:tasks.cephadm:Deploying osd.7 on smithi150 with /dev/vg_nvme/lv_2... 2023-07-14T18:07:49.859 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T18:07:49.902 INFO:journalctl@ceph.osd.6.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:07:50.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:49 smithi115 ceph-mon[127451]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:50.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:50.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:50.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:50.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:49 smithi150 ceph-mon[128853]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:50.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:50.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:50.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:50.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:49 smithi187 ceph-mon[129317]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:50.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:50.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:50.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:07:50.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:50.437 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:07:50.562 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:07:50 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:07:50.211+0000 7f5ba177d640 -1 osd.6 0 log_to_monitors true 2023-07-14T18:07:50.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:50 smithi115 ceph-mon[127451]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T18:07:50.890 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:50 smithi150 ceph-mon[128853]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T18:07:51.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:50 smithi187 ceph-mon[129317]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T18:07:51.799 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T18:07:51.799 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T18:07:51.799 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-07-14T18:07:51.799 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-07-14T18:07:51.800 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015214 s, 689 MB/s 2023-07-14T18:07:51.800 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-07-14T18:07:52.101 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:07:51 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:07:51.725+0000 7f5b96df5700 -1 osd.6 0 waiting for initial osdmap 2023-07-14T18:07:52.102 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:07:51 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:07:51.737+0000 7f5b8f3e4700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:07:52.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:51 smithi115 ceph-mon[127451]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T18:07:52.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:51 smithi115 ceph-mon[127451]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T18:07:52.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:52.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:51 smithi115 ceph-mon[127451]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:07:52.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:51 smithi115 ceph-mon[127451]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:52.116 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:51 smithi150 ceph-mon[128853]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T18:07:52.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:51 smithi150 ceph-mon[128853]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T18:07:52.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:52.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:51 smithi150 ceph-mon[128853]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:07:52.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:51 smithi150 ceph-mon[128853]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:52.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:51 smithi187 ceph-mon[129317]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T18:07:52.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:51 smithi187 ceph-mon[129317]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T18:07:52.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:52.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:51 smithi187 ceph-mon[129317]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:07:52.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:51 smithi187 ceph-mon[129317]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:52.195 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2023-07-14T18:07:52.751 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:07:53.094 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:07:53.095 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T18:07:53.095 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:53.095 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:53.095 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi115 2023-07-14T18:07:53.095 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.095 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.096 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:53.096 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:53.096 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:53.096 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi115 to 5722M 2023-07-14T18:07:53.096 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.096 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:53.096 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:53.097 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:52 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:07:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T18:07:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi115 2023-07-14T18:07:53.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:53.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:53.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:53.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi115 to 5722M 2023-07-14T18:07:53.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:53.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:53.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:52 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:07:53.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T18:07:53.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:53.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:53.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi115 2023-07-14T18:07:53.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:53.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:53.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:07:53.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi115 to 5722M 2023-07-14T18:07:53.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:53.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:07:53.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:52 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:07:53.840 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.805+0000 7f14e6a1b700 1 -- 172.21.15.150:0/627805172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 msgr2=0x7f14e01000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:53.840 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.805+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/627805172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 0x7f14e01000d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009990 tx=0x7f14d0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:07:53.840 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.805+0000 7f14e6a1b700 1 -- 172.21.15.150:0/627805172 shutdown_connections 2023-07-14T18:07:53.841 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.805+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/627805172 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14e0101850 0x7f14e0108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:53.841 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.805+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/627805172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 0x7f14e01000d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:53.841 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.805+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/627805172 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e0100ec0 0x7f14e0101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:53.841 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.805+0000 7f14e6a1b700 1 -- 172.21.15.150:0/627805172 >> 172.21.15.150:0/627805172 conn(0x7f14e00fb2d0 msgr2=0x7f14e00fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:07:53.841 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.805+0000 7f14e6a1b700 1 -- 172.21.15.150:0/627805172 shutdown_connections 2023-07-14T18:07:53.841 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.805+0000 7f14e6a1b700 1 -- 172.21.15.150:0/627805172 wait complete. 2023-07-14T18:07:53.841 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14e6a1b700 1 Processor -- start 2023-07-14T18:07:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14e6a1b700 1 -- start start 2023-07-14T18:07:53.843 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14e6a1b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 0x7f14e0195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:53.843 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14e6a1b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14e0100ec0 0x7f14e0195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:53.843 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14e6a1b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e0101850 0x7f14e019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:53.843 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14e6a1b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e019d4f0 con 0x7f14e0100ec0 2023-07-14T18:07:53.843 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14e6a1b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e019d630 con 0x7f14e00ffcc0 2023-07-14T18:07:53.844 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14e6a1b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e019d7c0 con 0x7f14e0101850 2023-07-14T18:07:53.844 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14dffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 0x7f14e0195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:53.844 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14dffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 0x7f14e0195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53652/0 (socket says 172.21.15.150:53652) 2023-07-14T18:07:53.844 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14dffff700 1 -- 172.21.15.150:0/702581140 learned_addr learned my addr 172.21.15.150:0/702581140 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-07-14T18:07:53.844 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14df7fe700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14e0100ec0 0x7f14e0195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:53.844 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14e4fb8700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e0101850 0x7f14e019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:53.844 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14dffff700 1 -- 172.21.15.150:0/702581140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e0101850 msgr2=0x7f14e019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:53.845 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14dffff700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e0101850 0x7f14e019cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:53.845 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14dffff700 1 -- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14e0100ec0 msgr2=0x7f14e0195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:07:53.845 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14dffff700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14e0100ec0 0x7f14e0195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:07:53.845 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.806+0000 7f14dffff700 1 -- 172.21.15.150:0/702581140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00095f0 con 0x7f14e00ffcc0 2023-07-14T18:07:53.845 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.807+0000 7f14e4fb8700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e0101850 0x7f14e019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:07:53.845 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.807+0000 7f14df7fe700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14e0100ec0 0x7f14e0195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:07:53.845 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.807+0000 7f14dffff700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 0x7f14e0195960 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f14d000c010 tx=0x7f14d0005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:07:53.845 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.807+0000 7f14dd7fa700 1 -- 172.21.15.150:0/702581140 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d001d070 con 0x7f14e00ffcc0 2023-07-14T18:07:53.846 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.807+0000 7f14dd7fa700 1 -- 172.21.15.150:0/702581140 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f14d0004100 con 0x7f14e00ffcc0 2023-07-14T18:07:53.846 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.807+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e019da40 con 0x7f14e00ffcc0 2023-07-14T18:07:53.846 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.807+0000 7f14dd7fa700 1 -- 172.21.15.150:0/702581140 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0005290 con 0x7f14e00ffcc0 2023-07-14T18:07:53.846 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.807+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e019de50 con 0x7f14e00ffcc0 2023-07-14T18:07:53.846 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.808+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e010b350 con 0x7f14e00ffcc0 2023-07-14T18:07:53.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.810+0000 7f14dd7fa700 1 -- 172.21.15.150:0/702581140 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f14d00047e0 con 0x7f14e00ffcc0 2023-07-14T18:07:53.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.810+0000 7f14dd7fa700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f14c80a1150 0x7f14c80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:07:53.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.810+0000 7f14dd7fa700 1 -- 172.21.15.150:0/702581140 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f14d00cd0a0 con 0x7f14e00ffcc0 2023-07-14T18:07:53.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.810+0000 7f14df7fe700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f14c80a1150 0x7f14c80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:07:53.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.811+0000 7f14df7fe700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f14c80a1150 0x7f14c80a3600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14d400bcf0 tx=0x7f14d400b2c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:07:53.848 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:53.814+0000 7f14dd7fa700 1 -- 172.21.15.150:0/702581140 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f14d0097590 con 0x7f14e00ffcc0 2023-07-14T18:07:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:07:54.010+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f14e0060a80 con 0x7f14c80a1150 2023-07-14T18:07:54.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:53 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:07:54.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:53 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:07:54.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:54.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:53 smithi115 ceph-mon[127451]: osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] boot 2023-07-14T18:07:54.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:53 smithi115 ceph-mon[127451]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T18:07:54.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:54.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:53 smithi115 ceph-mon[127451]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:54.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:53 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:07:54.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:53 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:07:54.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:54.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:53 smithi150 ceph-mon[128853]: osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] boot 2023-07-14T18:07:54.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:53 smithi150 ceph-mon[128853]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T18:07:54.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:54.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:53 smithi150 ceph-mon[128853]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:54.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:53 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:07:54.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:53 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:07:54.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:54.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:53 smithi187 ceph-mon[129317]: osd.6 [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] boot 2023-07-14T18:07:54.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:53 smithi187 ceph-mon[129317]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T18:07:54.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:07:54.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:53 smithi187 ceph-mon[129317]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:07:55.081 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:54 smithi150 ceph-mon[128853]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T18:07:55.082 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:54 smithi150 ceph-mon[128853]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:07:55.082 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:07:55.082 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:07:55.082 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:55.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:54 smithi115 ceph-mon[127451]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T18:07:55.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:54 smithi115 ceph-mon[127451]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:07:55.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:07:55.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:07:55.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:55.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:54 smithi187 ceph-mon[129317]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T18:07:55.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:54 smithi187 ceph-mon[129317]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:07:55.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:07:55.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:07:55.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:07:56.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:55 smithi115 ceph-mon[127451]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:07:56.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:55 smithi150 ceph-mon[128853]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:07:56.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:55 smithi187 ceph-mon[129317]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:07:57.101 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:56 smithi115 ceph-mon[127451]: from='client.? 172.21.15.150:0/2575064366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbe475b0-b4f9-41df-9615-a90c6fc62280"}]: dispatch 2023-07-14T18:07:57.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:56 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbe475b0-b4f9-41df-9615-a90c6fc62280"}]: dispatch 2023-07-14T18:07:57.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:56 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbe475b0-b4f9-41df-9615-a90c6fc62280"}]': finished 2023-07-14T18:07:57.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:56 smithi115 ceph-mon[127451]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T18:07:57.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:56 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:07:57.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:56 smithi150 ceph-mon[128853]: from='client.? 172.21.15.150:0/2575064366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbe475b0-b4f9-41df-9615-a90c6fc62280"}]: dispatch 2023-07-14T18:07:57.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:56 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbe475b0-b4f9-41df-9615-a90c6fc62280"}]: dispatch 2023-07-14T18:07:57.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:56 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbe475b0-b4f9-41df-9615-a90c6fc62280"}]': finished 2023-07-14T18:07:57.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:56 smithi150 ceph-mon[128853]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T18:07:57.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:56 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:07:57.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:56 smithi187 ceph-mon[129317]: from='client.? 172.21.15.150:0/2575064366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbe475b0-b4f9-41df-9615-a90c6fc62280"}]: dispatch 2023-07-14T18:07:57.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:56 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbe475b0-b4f9-41df-9615-a90c6fc62280"}]: dispatch 2023-07-14T18:07:57.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:56 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbe475b0-b4f9-41df-9615-a90c6fc62280"}]': finished 2023-07-14T18:07:57.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:56 smithi187 ceph-mon[129317]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T18:07:57.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:56 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:07:58.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:57 smithi115 ceph-mon[127451]: from='client.? 172.21.15.150:0/691038614' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:07:58.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:57 smithi115 ceph-mon[127451]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:07:58.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:57 smithi150 ceph-mon[128853]: from='client.? 172.21.15.150:0/691038614' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:07:58.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:57 smithi150 ceph-mon[128853]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:07:58.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:57 smithi187 ceph-mon[129317]: from='client.? 172.21.15.150:0/691038614' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:07:58.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:57 smithi187 ceph-mon[129317]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:00.308 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:07:59 smithi150 ceph-mon[128853]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:00.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:07:59 smithi115 ceph-mon[127451]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:00.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:07:59 smithi187 ceph-mon[129317]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:02.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:01 smithi150 ceph-mon[128853]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:02.351 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:01 smithi115 ceph-mon[127451]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:02.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:01 smithi187 ceph-mon[129317]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:03.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:03 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T18:08:03.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:03 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:03.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:03 smithi150 ceph-mon[128853]: Deploying daemon osd.7 on smithi150 2023-07-14T18:08:03.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:03 smithi150 ceph-mon[128853]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:03.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:03 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T18:08:03.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:03 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:03.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:03 smithi187 ceph-mon[129317]: Deploying daemon osd.7 on smithi150 2023-07-14T18:08:03.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:03 smithi187 ceph-mon[129317]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:03 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T18:08:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:03 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:03 smithi115 ceph-mon[127451]: Deploying daemon osd.7 on smithi150 2023-07-14T18:08:03.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:03 smithi115 ceph-mon[127451]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:06.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:05 smithi150 ceph-mon[128853]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:06.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:05 smithi115 ceph-mon[127451]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:06.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:05 smithi187 ceph-mon[129317]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:07.617 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:07.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:07.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:07 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:07.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:07 smithi150 ceph-mon[128853]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:07.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:07.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:07 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:07.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:07 smithi115 ceph-mon[127451]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:07.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:07.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:07.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:07 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:07.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:07 smithi187 ceph-mon[129317]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:09.100 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.065+0000 7f14dd7fa700 1 -- 172.21.15.150:0/702581140 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f14e0060a80 con 0x7f14c80a1150 2023-07-14T18:08:09.101 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 7 on host 'smithi150' 2023-07-14T18:08:09.103 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f14c80a1150 msgr2=0x7f14c80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:09.103 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f14c80a1150 0x7f14c80a3600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14d400bcf0 tx=0x7f14d400b2c0 comp rx=0 tx=0).stop 2023-07-14T18:08:09.103 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 msgr2=0x7f14e0195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 0x7f14e0195960 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f14d000c010 tx=0x7f14d0005ef0 comp rx=0 tx=0).stop 2023-07-14T18:08:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 shutdown_connections 2023-07-14T18:08:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f14c80a1150 0x7f14c80a3600 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14e0101850 0x7f14e019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e00ffcc0 0x7f14e0195960 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 --2- 172.21.15.150:0/702581140 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14e0100ec0 0x7f14e0195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 >> 172.21.15.150:0/702581140 conn(0x7f14e00fb2d0 msgr2=0x7f14e00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:08:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 shutdown_connections 2023-07-14T18:08:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:09.067+0000 7f14e6a1b700 1 -- 172.21.15.150:0/702581140 wait complete. 2023-07-14T18:08:09.446 DEBUG:teuthology.orchestra.run.smithi150:osd.7> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.7.service 2023-07-14T18:08:09.448 INFO:tasks.cephadm:Deploying osd.8 on smithi187 with /dev/vg_nvme/lv_2... 2023-07-14T18:08:09.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T18:08:09.481 INFO:journalctl@ceph.osd.7.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:08:10.021 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:08:10.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.065 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:09 smithi150 ceph-mon[128853]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:10.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.066 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:09 smithi115 ceph-mon[127451]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:10.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:09 smithi187 ceph-mon[129317]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:10.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:10.618 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:08:10 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:08:10.483+0000 7f86f6924640 -1 osd.7 0 log_to_monitors true 2023-07-14T18:08:10.905 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:10 smithi187 ceph-mon[129317]: from='osd.7 [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:08:10.905 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:10 smithi187 ceph-mon[129317]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:08:11.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:10 smithi115 ceph-mon[127451]: from='osd.7 [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:08:11.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:10 smithi115 ceph-mon[127451]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:08:11.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:10 smithi150 ceph-mon[128853]: from='osd.7 [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:08:11.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:10 smithi150 ceph-mon[128853]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:08:11.766 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T18:08:11.766 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T18:08:11.766 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2023-07-14T18:08:11.767 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2023-07-14T18:08:11.767 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150206 s, 698 MB/s 2023-07-14T18:08:11.767 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2023-07-14T18:08:11.832 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T18:08:11.833 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T18:08:11.833 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:11.833 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='osd.7 [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:08:11.833 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:08:11.833 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:11.833 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:11.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:11.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:11.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:11.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:11.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:11.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:11.834 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:11.835 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:12.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T18:08:12.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T18:08:12.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:12.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='osd.7 [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:08:12.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:08:12.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:12.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:12.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:12.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:12.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:12.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:12.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:12.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:12.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:12.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:12.117 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:08:11.795+0000 7f86ec79d700 -1 osd.7 0 waiting for initial osdmap 2023-07-14T18:08:12.118 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:08:11 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:08:11.803+0000 7f86e5d8e700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:08:12.168 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T18:08:12.169 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T18:08:12.169 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:12.169 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='osd.7 [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:08:12.169 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:08:12.169 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:08:12.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:12.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:12.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:12.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:12.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:12.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:12.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:12.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:12.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:12.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi187:vg_nvme/lv_2 2023-07-14T18:08:12.729 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:08:13.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi150 2023-07-14T18:08:13.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi150 to 5722M 2023-07-14T18:08:13.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:08:13.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T18:08:13.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:13.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:13.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:13.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:13.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:13.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:13.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: osd.7 [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] boot 2023-07-14T18:08:13.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T18:08:13.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:13.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi150 2023-07-14T18:08:13.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi150 to 5722M 2023-07-14T18:08:13.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:08:13.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T18:08:13.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:13.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:13.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:13.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:13.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:13.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:13.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: osd.7 [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] boot 2023-07-14T18:08:13.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T18:08:13.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:13.169 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi150 2023-07-14T18:08:13.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi150 to 5722M 2023-07-14T18:08:13.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:08:13.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T18:08:13.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:13.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:13.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:13.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:13.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:13.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:13.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: osd.7 [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] boot 2023-07-14T18:08:13.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T18:08:13.171 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:08:13.916 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.879+0000 7f01ab4ba700 1 -- 172.21.15.187:0/2550268258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 msgr2=0x7f01a4107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:13.917 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.879+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/2550268258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 0x7f01a4107000 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f019c00b0a0 tx=0x7f019c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:08:13.917 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.879+0000 7f01ab4ba700 1 -- 172.21.15.187:0/2550268258 shutdown_connections 2023-07-14T18:08:13.917 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.879+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/2550268258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 0x7f01a4107000 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:13.917 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.879+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/2550268258 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01a40febb0 0x7f01a40fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:13.917 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.879+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/2550268258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a40ffdb0 0x7f01a4100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:13.917 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.879+0000 7f01ab4ba700 1 -- 172.21.15.187:0/2550268258 >> 172.21.15.187:0/2550268258 conn(0x7f01a40fa1e0 msgr2=0x7f01a40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:08:13.917 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.879+0000 7f01ab4ba700 1 -- 172.21.15.187:0/2550268258 shutdown_connections 2023-07-14T18:08:13.918 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01ab4ba700 1 -- 172.21.15.187:0/2550268258 wait complete. 2023-07-14T18:08:13.918 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01ab4ba700 1 Processor -- start 2023-07-14T18:08:13.918 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01ab4ba700 1 -- start start 2023-07-14T18:08:13.918 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01ab4ba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a40febb0 0x7f01a41948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:13.918 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01ab4ba700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01a40ffdb0 0x7f01a4194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:13.918 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01ab4ba700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 0x7f01a419be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:13.918 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01ab4ba700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a419c430 con 0x7f01a40ffdb0 2023-07-14T18:08:13.919 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01ab4ba700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a419c570 con 0x7f01a40febb0 2023-07-14T18:08:13.919 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01ab4ba700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a419c700 con 0x7f01a4100740 2023-07-14T18:08:13.919 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01a9a57700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 0x7f01a419be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:13.919 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01a9a57700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 0x7f01a419be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:58670/0 (socket says 172.21.15.187:58670) 2023-07-14T18:08:13.919 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01a9a57700 1 -- 172.21.15.187:0/3083306114 learned_addr learned my addr 172.21.15.187:0/3083306114 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:08:13.919 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01a9256700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a40febb0 0x7f01a41948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:13.919 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.880+0000 7f01a8a55700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01a40ffdb0 0x7f01a4194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:13.919 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01a9a57700 1 -- 172.21.15.187:0/3083306114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a40febb0 msgr2=0x7f01a41948a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:13.920 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01a9a57700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a40febb0 0x7f01a41948a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:13.920 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01a9a57700 1 -- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01a40ffdb0 msgr2=0x7f01a4194de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:13.920 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01a9a57700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01a40ffdb0 0x7f01a4194de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:13.920 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01a9a57700 1 -- 172.21.15.187:0/3083306114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c009d00 con 0x7f01a4100740 2023-07-14T18:08:13.920 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01a9a57700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 0x7f01a419be60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f019c00d010 tx=0x7f019c004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:08:13.920 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01967fc700 1 -- 172.21.15.187:0/3083306114 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c00e040 con 0x7f01a4100740 2023-07-14T18:08:13.920 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01967fc700 1 -- 172.21.15.187:0/3083306114 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f019c01baf0 con 0x7f01a4100740 2023-07-14T18:08:13.921 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01967fc700 1 -- 172.21.15.187:0/3083306114 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c005790 con 0x7f01a4100740 2023-07-14T18:08:13.921 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a419c980 con 0x7f01a4100740 2023-07-14T18:08:13.921 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.881+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a419ce20 con 0x7f01a4100740 2023-07-14T18:08:13.922 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.882+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a4106130 con 0x7f01a4100740 2023-07-14T18:08:13.922 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.883+0000 7f01967fc700 1 -- 172.21.15.187:0/3083306114 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f019c01bc60 con 0x7f01a4100740 2023-07-14T18:08:13.922 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.883+0000 7f01967fc700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f01900a12a0 0x7f01900a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:13.922 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.884+0000 7f01967fc700 1 -- 172.21.15.187:0/3083306114 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f019c0cdf90 con 0x7f01a4100740 2023-07-14T18:08:13.922 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.884+0000 7f01a9256700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f01900a12a0 0x7f01900a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:13.922 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.884+0000 7f01a9256700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f01900a12a0 0x7f01900a3750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f01a40ffc10 tx=0x7f019800a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:08:13.923 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:13.887+0000 7f01967fc700 1 -- 172.21.15.187:0/3083306114 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f019c098330 con 0x7f01a4100740 2023-07-14T18:08:14.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:13 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:08:14.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:13 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:08:14.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:13 smithi115 ceph-mon[127451]: pgmap v152: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:14.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:13 smithi115 ceph-mon[127451]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T18:08:14.113 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:14.078+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f01a4060a80 con 0x7f01900a12a0 2023-07-14T18:08:14.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:13 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:08:14.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:13 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:08:14.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:13 smithi150 ceph-mon[128853]: pgmap v152: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:14.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:13 smithi150 ceph-mon[128853]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T18:08:14.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:13 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:08:14.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:13 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:08:14.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:13 smithi187 ceph-mon[129317]: pgmap v152: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:14.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:13 smithi187 ceph-mon[129317]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T18:08:15.101 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:14 smithi187 ceph-mon[129317]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:08:15.101 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:08:15.101 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:08:15.101 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:15.102 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:14 smithi187 ceph-mon[129317]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T18:08:15.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:14 smithi115 ceph-mon[127451]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:08:15.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:08:15.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:08:15.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:15.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:14 smithi115 ceph-mon[127451]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T18:08:15.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:14 smithi150 ceph-mon[128853]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:08:15.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:08:15.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:08:15.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:15.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:14 smithi150 ceph-mon[128853]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T18:08:16.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:15 smithi115 ceph-mon[127451]: pgmap v155: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:16.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:15 smithi115 ceph-mon[127451]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T18:08:16.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:15 smithi150 ceph-mon[128853]: pgmap v155: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:16.117 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:15 smithi150 ceph-mon[128853]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T18:08:16.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:15 smithi187 ceph-mon[129317]: pgmap v155: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:16.180 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:15 smithi187 ceph-mon[129317]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T18:08:17.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:17 smithi187 ceph-mon[129317]: osdmap e54: 8 total, 8 up, 8 in 2023-07-14T18:08:17.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:17 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/3669318430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574a1fb8-fa71-4c03-812d-bb1d8da70f8b"}]: dispatch 2023-07-14T18:08:17.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:17 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574a1fb8-fa71-4c03-812d-bb1d8da70f8b"}]: dispatch 2023-07-14T18:08:17.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:17 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "574a1fb8-fa71-4c03-812d-bb1d8da70f8b"}]': finished 2023-07-14T18:08:17.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:17 smithi187 ceph-mon[129317]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T18:08:17.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:17 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:17.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:17 smithi187 ceph-mon[129317]: pgmap v159: 1 pgs: 1 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:17.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:17 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/250301389' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:08:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:17 smithi115 ceph-mon[127451]: osdmap e54: 8 total, 8 up, 8 in 2023-07-14T18:08:17.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:17 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/3669318430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574a1fb8-fa71-4c03-812d-bb1d8da70f8b"}]: dispatch 2023-07-14T18:08:17.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:17 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574a1fb8-fa71-4c03-812d-bb1d8da70f8b"}]: dispatch 2023-07-14T18:08:17.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:17 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "574a1fb8-fa71-4c03-812d-bb1d8da70f8b"}]': finished 2023-07-14T18:08:17.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:17 smithi115 ceph-mon[127451]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T18:08:17.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:17 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:17.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:17 smithi115 ceph-mon[127451]: pgmap v159: 1 pgs: 1 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:17.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:17 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/250301389' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:08:17.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:17 smithi150 ceph-mon[128853]: osdmap e54: 8 total, 8 up, 8 in 2023-07-14T18:08:17.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:17 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/3669318430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574a1fb8-fa71-4c03-812d-bb1d8da70f8b"}]: dispatch 2023-07-14T18:08:17.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:17 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574a1fb8-fa71-4c03-812d-bb1d8da70f8b"}]: dispatch 2023-07-14T18:08:17.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:17 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "574a1fb8-fa71-4c03-812d-bb1d8da70f8b"}]': finished 2023-07-14T18:08:17.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:17 smithi150 ceph-mon[128853]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T18:08:17.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:17 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:17.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:17 smithi150 ceph-mon[128853]: pgmap v159: 1 pgs: 1 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:17.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:17 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/250301389' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:08:20.249 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:19 smithi187 ceph-mon[129317]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:20.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:19 smithi115 ceph-mon[127451]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:20.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:19 smithi150 ceph-mon[128853]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:22.087 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:21 smithi187 ceph-mon[129317]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:22.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:21 smithi115 ceph-mon[127451]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:22.367 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:21 smithi150 ceph-mon[128853]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:23.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:23 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T18:08:23.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:23 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:23.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:23 smithi187 ceph-mon[129317]: Deploying daemon osd.8 on smithi187 2023-07-14T18:08:23.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:23 smithi187 ceph-mon[129317]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:23 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T18:08:23.852 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:23 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:23.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:23 smithi115 ceph-mon[127451]: Deploying daemon osd.8 on smithi187 2023-07-14T18:08:23.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:23 smithi115 ceph-mon[127451]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:23.867 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:23 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T18:08:23.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:23 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:23.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:23 smithi150 ceph-mon[128853]: Deploying daemon osd.8 on smithi187 2023-07-14T18:08:23.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:23 smithi150 ceph-mon[128853]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:26.265 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:25 smithi187 ceph-mon[129317]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:26.352 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:25 smithi115 ceph-mon[127451]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:26.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:25 smithi150 ceph-mon[128853]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:28.323 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:28 smithi187 ceph-mon[129317]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:28.323 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:28.324 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:28.324 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:28 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:28 smithi115 ceph-mon[127451]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:28.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:28 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:28.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:28 smithi150 ceph-mon[128853]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:28.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:28.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:28.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:28 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:30.022 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.986+0000 7f01967fc700 1 -- 172.21.15.187:0/3083306114 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f01a4060a80 con 0x7f01900a12a0 2023-07-14T18:08:30.023 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 8 on host 'smithi187' 2023-07-14T18:08:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f01900a12a0 msgr2=0x7f01900a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f01900a12a0 0x7f01900a3750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f01a40ffc10 tx=0x7f019800a3b0 comp rx=0 tx=0).stop 2023-07-14T18:08:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 msgr2=0x7f01a419be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 0x7f01a419be60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f019c00d010 tx=0x7f019c004d00 comp rx=0 tx=0).stop 2023-07-14T18:08:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 shutdown_connections 2023-07-14T18:08:30.024 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f01900a12a0 0x7f01900a3750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a4100740 0x7f01a419be60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a40febb0 0x7f01a41948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 --2- 172.21.15.187:0/3083306114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01a40ffdb0 0x7f01a4194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 >> 172.21.15.187:0/3083306114 conn(0x7f01a40fa1e0 msgr2=0x7f01a4103270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:08:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 shutdown_connections 2023-07-14T18:08:30.025 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:08:29.988+0000 7f01ab4ba700 1 -- 172.21.15.187:0/3083306114 wait complete. 2023-07-14T18:08:30.417 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:30 smithi187 ceph-mon[129317]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:30.417 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.417 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.417 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:30.417 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:30.417 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.418 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.418 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.418 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:30.419 DEBUG:teuthology.orchestra.run.smithi187:osd.8> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.8.service 2023-07-14T18:08:30.421 INFO:tasks.cephadm:Deploying osd.9 on smithi115 with /dev/vg_nvme/lv_1... 2023-07-14T18:08:30.421 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T18:08:30.452 INFO:journalctl@ceph.osd.8.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:08:30.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:30 smithi115 ceph-mon[127451]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:30.453 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:30.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:30.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:30.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:30 smithi150 ceph-mon[128853]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:30.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:30.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:30.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:30.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:30.986 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:08:31.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:31 smithi187 ceph-mon[129317]: from='osd.8 [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:08:31.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:31 smithi187 ceph-mon[129317]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:08:31.315 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:08:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:08:31.043+0000 7fcb7d0e2640 -1 osd.8 0 log_to_monitors true 2023-07-14T18:08:31.417 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:31 smithi115 ceph-mon[127451]: from='osd.8 [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:08:31.417 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:31 smithi115 ceph-mon[127451]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:08:31.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:31 smithi150 ceph-mon[128853]: from='osd.8 [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:08:31.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:31 smithi150 ceph-mon[128853]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:08:32.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:32 smithi187 ceph-mon[129317]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:32.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:32 smithi187 ceph-mon[129317]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T18:08:32.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:32 smithi187 ceph-mon[129317]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T18:08:32.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:32 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:32.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:32 smithi187 ceph-mon[129317]: from='osd.8 [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:08:32.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:32 smithi187 ceph-mon[129317]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:08:32.432 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:08:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:08:32.136+0000 7fcb7275a700 -1 osd.8 0 waiting for initial osdmap 2023-07-14T18:08:32.432 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:08:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:08:32.145+0000 7fcb6c54c700 -1 osd.8 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:08:32.602 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T18:08:32.603 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T18:08:32.603 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2023-07-14T18:08:32.603 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2023-07-14T18:08:32.603 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112345 s, 933 MB/s 2023-07-14T18:08:32.603 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2023-07-14T18:08:32.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:32 smithi115 ceph-mon[127451]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:32.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:32 smithi115 ceph-mon[127451]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T18:08:32.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:32 smithi115 ceph-mon[127451]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T18:08:32.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:32 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:32.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:32 smithi115 ceph-mon[127451]: from='osd.8 [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:08:32.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:32 smithi115 ceph-mon[127451]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:08:32.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:32 smithi150 ceph-mon[128853]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:32.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:32 smithi150 ceph-mon[128853]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T18:08:32.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:32 smithi150 ceph-mon[128853]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T18:08:32.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:32 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:32.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:32 smithi150 ceph-mon[128853]: from='osd.8 [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:08:32.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:32 smithi150 ceph-mon[128853]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:08:33.040 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_1 2023-07-14T18:08:33.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:08:33.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: osdmap e57: 9 total, 8 up, 9 in 2023-07-14T18:08:33.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:33.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:33.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: osd.8 [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] boot 2023-07-14T18:08:33.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T18:08:33.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:33.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:33.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:33.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:33.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:33.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:33.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:08:33.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: osdmap e57: 9 total, 8 up, 9 in 2023-07-14T18:08:33.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:33.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:33.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: osd.8 [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] boot 2023-07-14T18:08:33.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T18:08:33.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:33.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:33.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:33.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:33.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:33.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:33.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:08:33.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: osdmap e57: 9 total, 8 up, 9 in 2023-07-14T18:08:33.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:33.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:33.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: osd.8 [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] boot 2023-07-14T18:08:33.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T18:08:33.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:08:33.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:33.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:33.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:33.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:33.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:33.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:33.636 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:08:34.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:34 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:08:34.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:34 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:08:34.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:34 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi187 2023-07-14T18:08:34.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:34 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi187 to 5722M 2023-07-14T18:08:34.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:34 smithi187 ceph-mon[129317]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:34.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:34 smithi187 ceph-mon[129317]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T18:08:34.602 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:34 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:08:34.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:34 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:08:34.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:34 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi187 2023-07-14T18:08:34.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:34 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi187 to 5722M 2023-07-14T18:08:34.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:34 smithi115 ceph-mon[127451]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:34.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:34 smithi115 ceph-mon[127451]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T18:08:34.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:34 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:08:34.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:34 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:08:34.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:34 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi187 2023-07-14T18:08:34.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:34 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi187 to 5722M 2023-07-14T18:08:34.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:34 smithi150 ceph-mon[128853]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:08:34.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:34 smithi150 ceph-mon[128853]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T18:08:34.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.806+0000 7fcf4c44a700 1 -- 172.21.15.115:0/3044339333 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440fe340 msgr2=0x7fcf440fe750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.806+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/3044339333 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440fe340 0x7fcf440fe750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009990 tx=0x7fcf34009ca0 comp rx=0 tx=0).stop 2023-07-14T18:08:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 -- 172.21.15.115:0/3044339333 shutdown_connections 2023-07-14T18:08:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/3044339333 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf440ffed0 0x7fcf44106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/3044339333 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440fe340 0x7fcf440fe750 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:34.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/3044339333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf440ff540 0x7fcf440ff990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 -- 172.21.15.115:0/3044339333 >> 172.21.15.115:0/3044339333 conn(0x7fcf440f9990 msgr2=0x7fcf440fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:08:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 -- 172.21.15.115:0/3044339333 shutdown_connections 2023-07-14T18:08:34.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 -- 172.21.15.115:0/3044339333 wait complete. 2023-07-14T18:08:34.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 Processor -- start 2023-07-14T18:08:34.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 -- start start 2023-07-14T18:08:34.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf440fe340 0x7fcf441033e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:34.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440ff540 0x7fcf44101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:34.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf440ffed0 0x7fcf44101f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:34.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf441026f0 con 0x7fcf440ff540 2023-07-14T18:08:34.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44102860 con 0x7fcf440ffed0 2023-07-14T18:08:34.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.807+0000 7fcf4c44a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4419f920 con 0x7fcf440fe340 2023-07-14T18:08:34.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf499e5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440ff540 0x7fcf44101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:34.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf499e5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440ff540 0x7fcf44101a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:51994/0 (socket says 172.21.15.115:51994) 2023-07-14T18:08:34.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf499e5700 1 -- 172.21.15.115:0/533430039 learned_addr learned my addr 172.21.15.115:0/533430039 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:08:34.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf4a1e6700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf440fe340 0x7fcf441033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:34.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf4a9e7700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf440ffed0 0x7fcf44101f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:34.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf499e5700 1 -- 172.21.15.115:0/533430039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf440fe340 msgr2=0x7fcf441033e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:34.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf499e5700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf440fe340 0x7fcf441033e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:34.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf499e5700 1 -- 172.21.15.115:0/533430039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf440ffed0 msgr2=0x7fcf44101f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:34.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf499e5700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf440ffed0 0x7fcf44101f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:34.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf499e5700 1 -- 172.21.15.115:0/533430039 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf340095f0 con 0x7fcf440ff540 2023-07-14T18:08:34.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf499e5700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440ff540 0x7fcf44101a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000c920 tx=0x7fcf4000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:08:34.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf3b7fe700 1 -- 172.21.15.115:0/533430039 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf40004a30 con 0x7fcf440ff540 2023-07-14T18:08:34.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf3b7fe700 1 -- 172.21.15.115:0/533430039 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcf400054b0 con 0x7fcf440ff540 2023-07-14T18:08:34.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf4419fc00 con 0x7fcf440ff540 2023-07-14T18:08:34.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf3b7fe700 1 -- 172.21.15.115:0/533430039 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf40007b50 con 0x7fcf440ff540 2023-07-14T18:08:34.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.808+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf441a0100 con 0x7fcf440ff540 2023-07-14T18:08:34.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.809+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf441099d0 con 0x7fcf440ff540 2023-07-14T18:08:34.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.810+0000 7fcf3b7fe700 1 -- 172.21.15.115:0/533430039 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fcf40007cb0 con 0x7fcf440ff540 2023-07-14T18:08:34.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.811+0000 7fcf3b7fe700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcf300a1150 0x7fcf300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:34.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.811+0000 7fcf4a1e6700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcf300a1150 0x7fcf300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:34.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.811+0000 7fcf3b7fe700 1 -- 172.21.15.115:0/533430039 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fcf400cba70 con 0x7fcf440ff540 2023-07-14T18:08:34.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.811+0000 7fcf4a1e6700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcf300a1150 0x7fcf300a3600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcf3400b5c0 tx=0x7fcf34005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:08:34.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:34.815+0000 7fcf3b7fe700 1 -- 172.21.15.115:0/533430039 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcf40095b50 con 0x7fcf440ff540 2023-07-14T18:08:35.045 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:35.009+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf44105a90 con 0x7fcf300a1150 2023-07-14T18:08:35.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:35 smithi187 ceph-mon[129317]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T18:08:35.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:35 smithi187 ceph-mon[129317]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:35.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:35 smithi187 ceph-mon[129317]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:08:35.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:08:35.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:08:35.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:35.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:35 smithi115 ceph-mon[127451]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T18:08:35.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:35 smithi115 ceph-mon[127451]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:35.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:35 smithi115 ceph-mon[127451]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:08:35.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:08:35.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:08:35.851 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:35.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:35 smithi150 ceph-mon[128853]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T18:08:35.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:35 smithi150 ceph-mon[128853]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:35.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:35 smithi150 ceph-mon[128853]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:08:35.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:08:35.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:08:35.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:36.698 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:36 smithi115 ceph-mon[127451]: osdmap e61: 9 total, 9 up, 9 in 2023-07-14T18:08:36.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:36 smithi150 ceph-mon[128853]: osdmap e61: 9 total, 9 up, 9 in 2023-07-14T18:08:36.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:36 smithi187 ceph-mon[129317]: osdmap e61: 9 total, 9 up, 9 in 2023-07-14T18:08:37.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:37 smithi115 ceph-mon[127451]: osdmap e62: 9 total, 9 up, 9 in 2023-07-14T18:08:37.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:37 smithi115 ceph-mon[127451]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:37.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:37 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/522843103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd79ae74-eaa1-4b63-aad8-c77864ba2eed"}]: dispatch 2023-07-14T18:08:37.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:37 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/522843103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd79ae74-eaa1-4b63-aad8-c77864ba2eed"}]': finished 2023-07-14T18:08:37.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:37 smithi115 ceph-mon[127451]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T18:08:37.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:37 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:37.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:37 smithi150 ceph-mon[128853]: osdmap e62: 9 total, 9 up, 9 in 2023-07-14T18:08:37.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:37 smithi150 ceph-mon[128853]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:37.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:37 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/522843103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd79ae74-eaa1-4b63-aad8-c77864ba2eed"}]: dispatch 2023-07-14T18:08:37.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:37 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/522843103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd79ae74-eaa1-4b63-aad8-c77864ba2eed"}]': finished 2023-07-14T18:08:37.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:37 smithi150 ceph-mon[128853]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T18:08:37.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:37 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:37.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:37 smithi187 ceph-mon[129317]: osdmap e62: 9 total, 9 up, 9 in 2023-07-14T18:08:37.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:37 smithi187 ceph-mon[129317]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:37.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:37 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/522843103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd79ae74-eaa1-4b63-aad8-c77864ba2eed"}]: dispatch 2023-07-14T18:08:37.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:37 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/522843103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd79ae74-eaa1-4b63-aad8-c77864ba2eed"}]': finished 2023-07-14T18:08:37.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:37 smithi187 ceph-mon[129317]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T18:08:37.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:37 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:38.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:38 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1908073730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:08:38.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:38 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1908073730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:08:38.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:38 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1908073730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:08:39.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:39 smithi115 ceph-mon[127451]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-14T18:08:39.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:39 smithi150 ceph-mon[128853]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-14T18:08:39.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:39 smithi187 ceph-mon[129317]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-14T18:08:42.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:41 smithi115 ceph-mon[127451]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:08:42.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:41 smithi150 ceph-mon[128853]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:08:42.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:41 smithi187 ceph-mon[129317]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:08:43.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:43 smithi115 ceph-mon[127451]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:08:43.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:43 smithi187 ceph-mon[129317]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:08:43.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:43 smithi150 ceph-mon[128853]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:08:44.680 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:44 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T18:08:44.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:44 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:44.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:44 smithi187 ceph-mon[129317]: Deploying daemon osd.9 on smithi115 2023-07-14T18:08:44.735 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:44 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T18:08:44.736 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:44 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:44.736 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:44 smithi115 ceph-mon[127451]: Deploying daemon osd.9 on smithi115 2023-07-14T18:08:44.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:44 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T18:08:44.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:44 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:44.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:44 smithi150 ceph-mon[128853]: Deploying daemon osd.9 on smithi115 2023-07-14T18:08:45.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:45 smithi115 ceph-mon[127451]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-14T18:08:45.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:45 smithi150 ceph-mon[128853]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-14T18:08:45.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:45 smithi187 ceph-mon[129317]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-14T18:08:48.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:47 smithi115 ceph-mon[127451]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T18:08:48.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:47 smithi150 ceph-mon[128853]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T18:08:48.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:47 smithi187 ceph-mon[129317]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T18:08:49.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:49.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:49.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:49 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:49.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:49 smithi115 ceph-mon[127451]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-14T18:08:49.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:49.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:49.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:49 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:49.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:49 smithi187 ceph-mon[129317]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-14T18:08:49.868 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:49.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:49.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:49 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:49.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:49 smithi150 ceph-mon[128853]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-14T18:08:51.193 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 9 on host 'smithi115' 2023-07-14T18:08:51.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.158+0000 7fcf3b7fe700 1 -- 172.21.15.115:0/533430039 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcf44105a90 con 0x7fcf300a1150 2023-07-14T18:08:51.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcf300a1150 msgr2=0x7fcf300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:51.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcf300a1150 0x7fcf300a3600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcf3400b5c0 tx=0x7fcf34005fb0 comp rx=0 tx=0).stop 2023-07-14T18:08:51.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440ff540 msgr2=0x7fcf44101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:51.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440ff540 0x7fcf44101a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000c920 tx=0x7fcf4000cc30 comp rx=0 tx=0).stop 2023-07-14T18:08:51.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 shutdown_connections 2023-07-14T18:08:51.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcf300a1150 0x7fcf300a3600 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:51.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf440ffed0 0x7fcf44101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:51.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf440fe340 0x7fcf441033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:51.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 --2- 172.21.15.115:0/533430039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf440ff540 0x7fcf44101a30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:51.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 >> 172.21.15.115:0/533430039 conn(0x7fcf440f9990 msgr2=0x7fcf440fbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:08:51.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 shutdown_connections 2023-07-14T18:08:51.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:08:51.160+0000 7fcf4c44a700 1 -- 172.21.15.115:0/533430039 wait complete. 2023-07-14T18:08:51.614 DEBUG:teuthology.orchestra.run.smithi115:osd.9> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.9.service 2023-07-14T18:08:51.616 INFO:tasks.cephadm:Deploying osd.10 on smithi150 with /dev/vg_nvme/lv_1... 2023-07-14T18:08:51.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T18:08:51.646 INFO:journalctl@ceph.osd.9.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:08:52.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:51 smithi187 ceph-mon[129317]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:52.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:51 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.182 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:08:52.231 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:51 smithi115 ceph-mon[127451]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:52.231 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:51 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:51 smithi150 ceph-mon[128853]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:52.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:51 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:52.559 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:08:52 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:08:52.253+0000 7fae42714640 -1 osd.9 0 log_to_monitors true 2023-07-14T18:08:53.086 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:52 smithi115 ceph-mon[127451]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T18:08:53.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:52 smithi187 ceph-mon[129317]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T18:08:53.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:52 smithi150 ceph-mon[128853]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T18:08:53.548 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T18:08:53.548 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T18:08:53.548 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-07-14T18:08:53.548 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-07-14T18:08:53.548 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011032 s, 950 MB/s 2023-07-14T18:08:53.549 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-07-14T18:08:53.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi150:vg_nvme/lv_1 2023-07-14T18:08:54.353 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:08:53.897+0000 7fae37d8c700 -1 osd.9 0 waiting for initial osdmap 2023-07-14T18:08:54.353 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:08:53.916+0000 7fae31b7e700 -1 osd.9 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:08:54.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:54.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T18:08:54.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: osdmap e64: 10 total, 9 up, 10 in 2023-07-14T18:08:54.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:54.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:08:54.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:54.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:54.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:54.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T18:08:54.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: osdmap e64: 10 total, 9 up, 10 in 2023-07-14T18:08:54.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:54.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:08:54.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:54.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:54.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:54.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T18:08:54.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: osdmap e64: 10 total, 9 up, 10 in 2023-07-14T18:08:54.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:54.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2023-07-14T18:08:54.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:54.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:54.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:08:54.488 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:08:54.951 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi115 2023-07-14T18:08:54.951 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi115 to 4292M 2023-07-14T18:08:54.951 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:54.951 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:54.951 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:54.952 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:08:54.952 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: osdmap e65: 10 total, 9 up, 10 in 2023-07-14T18:08:54.952 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:54.952 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:54.952 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:54.952 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:54.952 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:54.953 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:54.953 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:54 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:55.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi115 2023-07-14T18:08:55.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi115 to 4292M 2023-07-14T18:08:55.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:55.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:55.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:55.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:08:55.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: osdmap e65: 10 total, 9 up, 10 in 2023-07-14T18:08:55.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:55.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:55.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:55.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:55.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:55.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:55.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:54 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:55.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi115 2023-07-14T18:08:55.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi115 to 4292M 2023-07-14T18:08:55.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:55.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:55.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:55.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2023-07-14T18:08:55.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: osdmap e65: 10 total, 9 up, 10 in 2023-07-14T18:08:55.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:55.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:55.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:55.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:08:55.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:55.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:08:55.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:54 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:08:55.530 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.494+0000 7f5f42090700 1 -- 172.21.15.150:0/1791397584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c100170 msgr2=0x7f5f3c100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:55.530 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.494+0000 7f5f42090700 1 --2- 172.21.15.150:0/1791397584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c100170 0x7f5f3c100580 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5f300099c0 tx=0x7f5f30009cd0 comp rx=0 tx=0).stop 2023-07-14T18:08:55.530 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.494+0000 7f5f42090700 1 -- 172.21.15.150:0/1791397584 shutdown_connections 2023-07-14T18:08:55.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.494+0000 7f5f42090700 1 --2- 172.21.15.150:0/1791397584 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f3c0ff780 0x7f5f3c107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:55.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.494+0000 7f5f42090700 1 --2- 172.21.15.150:0/1791397584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c100170 0x7f5f3c100580 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:55.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.494+0000 7f5f42090700 1 --2- 172.21.15.150:0/1791397584 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f3c0fedd0 0x7f5f3c0ff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:55.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.494+0000 7f5f42090700 1 -- 172.21.15.150:0/1791397584 >> 172.21.15.150:0/1791397584 conn(0x7f5f3c0f9990 msgr2=0x7f5f3c0fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:08:55.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.494+0000 7f5f42090700 1 -- 172.21.15.150:0/1791397584 shutdown_connections 2023-07-14T18:08:55.531 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.494+0000 7f5f42090700 1 -- 172.21.15.150:0/1791397584 wait complete. 2023-07-14T18:08:55.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f42090700 1 Processor -- start 2023-07-14T18:08:55.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f42090700 1 -- start start 2023-07-14T18:08:55.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f42090700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f3c0fedd0 0x7f5f3c1948c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:55.534 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f42090700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c0ff780 0x7f5f3c194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:55.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f42090700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f3c100170 0x7f5f3c19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:55.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f42090700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c19c450 con 0x7f5f3c0fedd0 2023-07-14T18:08:55.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f42090700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c19c590 con 0x7f5f3c0ff780 2023-07-14T18:08:55.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f42090700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c19c720 con 0x7f5f3c100170 2023-07-14T18:08:55.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f3affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c0ff780 0x7f5f3c194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:55.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f3affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c0ff780 0x7f5f3c194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53800/0 (socket says 172.21.15.150:53800) 2023-07-14T18:08:55.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f3affd700 1 -- 172.21.15.150:0/2025089529 learned_addr learned my addr 172.21.15.150:0/2025089529 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-07-14T18:08:55.535 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f3bfff700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f3c100170 0x7f5f3c19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:55.536 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f3b7fe700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f3c0fedd0 0x7f5f3c1948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:55.536 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.495+0000 7f5f3affd700 1 -- 172.21.15.150:0/2025089529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f3c100170 msgr2=0x7f5f3c19be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:55.536 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f3affd700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f3c100170 0x7f5f3c19be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:55.536 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f3affd700 1 -- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f3c0fedd0 msgr2=0x7f5f3c1948c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:08:55.536 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f3affd700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f3c0fedd0 0x7f5f3c1948c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:08:55.536 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f3affd700 1 -- 172.21.15.150:0/2025089529 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f300095f0 con 0x7f5f3c0ff780 2023-07-14T18:08:55.536 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f3affd700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c0ff780 0x7f5f3c194e00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00b530 tx=0x7f5f2c00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:08:55.537 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f38ff9700 1 -- 172.21.15.150:0/2025089529 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c011830 con 0x7f5f3c0ff780 2023-07-14T18:08:55.537 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f38ff9700 1 -- 172.21.15.150:0/2025089529 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5f2c011e20 con 0x7f5f3c0ff780 2023-07-14T18:08:55.537 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3c19ca00 con 0x7f5f3c0ff780 2023-07-14T18:08:55.537 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f38ff9700 1 -- 172.21.15.150:0/2025089529 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c00f2e0 con 0x7f5f3c0ff780 2023-07-14T18:08:55.537 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.496+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3c19cf00 con 0x7f5f3c0ff780 2023-07-14T18:08:55.537 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.497+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3c10a300 con 0x7f5f3c0ff780 2023-07-14T18:08:55.537 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.498+0000 7f5f38ff9700 1 -- 172.21.15.150:0/2025089529 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f5f2c011990 con 0x7f5f3c0ff780 2023-07-14T18:08:55.538 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.498+0000 7f5f38ff9700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5f240a1150 0x7f5f240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:08:55.538 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.499+0000 7f5f38ff9700 1 -- 172.21.15.150:0/2025089529 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f5f2c0cc040 con 0x7f5f3c0ff780 2023-07-14T18:08:55.538 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.499+0000 7f5f3b7fe700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5f240a1150 0x7f5f240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:08:55.539 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.499+0000 7f5f3b7fe700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5f240a1150 0x7f5f240a3600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5f30009f50 tx=0x7f5f3000b790 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:08:55.539 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.502+0000 7f5f38ff9700 1 -- 172.21.15.150:0/2025089529 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5f2c095f90 con 0x7f5f3c0ff780 2023-07-14T18:08:55.730 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:08:55.695+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f3c060a80 con 0x7f5f240a1150 2023-07-14T18:08:56.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:08:56.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:08:56.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:56.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:56.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] boot 2023-07-14T18:08:56.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: osdmap e66: 10 total, 10 up, 10 in 2023-07-14T18:08:56.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:56.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:08:56.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:08:56.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:56.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:08:56.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:08:56.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:56.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:56.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] boot 2023-07-14T18:08:56.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: osdmap e66: 10 total, 10 up, 10 in 2023-07-14T18:08:56.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:56.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:08:56.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:08:56.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:56.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:08:56.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:08:56.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:08:56.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:56.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: osd.9 [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] boot 2023-07-14T18:08:56.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: osdmap e66: 10 total, 10 up, 10 in 2023-07-14T18:08:56.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:08:56.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:08:56.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:08:56.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:08:57.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:57 smithi187 ceph-mon[129317]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:08:57.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:57 smithi187 ceph-mon[129317]: osdmap e67: 10 total, 10 up, 10 in 2023-07-14T18:08:57.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:57 smithi115 ceph-mon[127451]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:08:57.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:57 smithi115 ceph-mon[127451]: osdmap e67: 10 total, 10 up, 10 in 2023-07-14T18:08:57.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:57 smithi150 ceph-mon[128853]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:08:57.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:57 smithi150 ceph-mon[128853]: osdmap e67: 10 total, 10 up, 10 in 2023-07-14T18:08:58.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:58 smithi187 ceph-mon[129317]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:08:58.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:58 smithi187 ceph-mon[129317]: from='client.? 172.21.15.150:0/166314609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d66bdb3-b20d-4474-ab4b-9d0f81232272"}]: dispatch 2023-07-14T18:08:58.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:58 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d66bdb3-b20d-4474-ab4b-9d0f81232272"}]: dispatch 2023-07-14T18:08:58.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:58 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d66bdb3-b20d-4474-ab4b-9d0f81232272"}]': finished 2023-07-14T18:08:58.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:58 smithi187 ceph-mon[129317]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T18:08:58.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:58 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:08:58.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:58 smithi115 ceph-mon[127451]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:08:58.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:58 smithi115 ceph-mon[127451]: from='client.? 172.21.15.150:0/166314609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d66bdb3-b20d-4474-ab4b-9d0f81232272"}]: dispatch 2023-07-14T18:08:58.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:58 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d66bdb3-b20d-4474-ab4b-9d0f81232272"}]: dispatch 2023-07-14T18:08:58.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:58 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d66bdb3-b20d-4474-ab4b-9d0f81232272"}]': finished 2023-07-14T18:08:58.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:58 smithi115 ceph-mon[127451]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T18:08:58.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:58 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:08:58.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:58 smithi150 ceph-mon[128853]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:08:58.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:58 smithi150 ceph-mon[128853]: from='client.? 172.21.15.150:0/166314609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d66bdb3-b20d-4474-ab4b-9d0f81232272"}]: dispatch 2023-07-14T18:08:58.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:58 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d66bdb3-b20d-4474-ab4b-9d0f81232272"}]: dispatch 2023-07-14T18:08:58.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:58 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d66bdb3-b20d-4474-ab4b-9d0f81232272"}]': finished 2023-07-14T18:08:58.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:58 smithi150 ceph-mon[128853]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T18:08:58.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:58 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:08:59.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:08:59 smithi187 ceph-mon[129317]: from='client.? 172.21.15.150:0/1662724089' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:08:59.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:08:59 smithi115 ceph-mon[127451]: from='client.? 172.21.15.150:0/1662724089' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:08:59.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:08:59 smithi150 ceph-mon[128853]: from='client.? 172.21.15.150:0/1662724089' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:09:00.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:00 smithi187 ceph-mon[129317]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:00.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:00 smithi115 ceph-mon[127451]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:00.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:00 smithi150 ceph-mon[128853]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:02.368 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:02 smithi150 ceph-mon[128853]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:02.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:02 smithi187 ceph-mon[129317]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:02.603 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:02 smithi115 ceph-mon[127451]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:03.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:03 smithi115 ceph-mon[127451]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:03.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:03 smithi150 ceph-mon[128853]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:03.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:03 smithi187 ceph-mon[129317]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:04.618 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:04 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T18:09:04.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:04 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:04.619 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:04 smithi150 ceph-mon[128853]: Deploying daemon osd.10 on smithi150 2023-07-14T18:09:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:04 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T18:09:04.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:04 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:04.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:04 smithi115 ceph-mon[127451]: Deploying daemon osd.10 on smithi150 2023-07-14T18:09:04.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T18:09:04.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:04 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:04.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:04 smithi187 ceph-mon[129317]: Deploying daemon osd.10 on smithi150 2023-07-14T18:09:05.707 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:05 smithi150 ceph-mon[128853]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:05.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:05 smithi115 ceph-mon[127451]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:05.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:05 smithi187 ceph-mon[129317]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:08.274 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:07 smithi150 ceph-mon[128853]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:08.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:07 smithi115 ceph-mon[127451]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:08.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:07 smithi187 ceph-mon[129317]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:09.574 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:09.574 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:09.574 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:09 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:09.574 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:09 smithi150 ceph-mon[128853]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:09.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:09.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:09.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:09 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:09.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:09 smithi115 ceph-mon[127451]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:09.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:09.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:09.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:09 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:09.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:09 smithi187 ceph-mon[129317]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:10.994 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 10 on host 'smithi150' 2023-07-14T18:09:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.956+0000 7f5f38ff9700 1 -- 172.21.15.150:0/2025089529 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f5f3c060a80 con 0x7f5f240a1150 2023-07-14T18:09:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5f240a1150 msgr2=0x7f5f240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5f240a1150 0x7f5f240a3600 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5f30009f50 tx=0x7f5f3000b790 comp rx=0 tx=0).stop 2023-07-14T18:09:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c0ff780 msgr2=0x7f5f3c194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c0ff780 0x7f5f3c194e00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00b530 tx=0x7f5f2c00b840 comp rx=0 tx=0).stop 2023-07-14T18:09:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 shutdown_connections 2023-07-14T18:09:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5f240a1150 0x7f5f240a3600 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f3c100170 0x7f5f3c19be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5f3c0fedd0 0x7f5f3c1948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 --2- 172.21.15.150:0/2025089529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f3c0ff780 0x7f5f3c194e00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 >> 172.21.15.150:0/2025089529 conn(0x7f5f3c0f9990 msgr2=0x7f5f3c0fbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 shutdown_connections 2023-07-14T18:09:10.998 INFO:teuthology.orchestra.run.smithi150.stderr:2023-07-14T18:09:10.959+0000 7f5f42090700 1 -- 172.21.15.150:0/2025089529 wait complete. 2023-07-14T18:09:11.365 DEBUG:teuthology.orchestra.run.smithi150:osd.10> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.10.service 2023-07-14T18:09:11.367 INFO:tasks.cephadm:Deploying osd.11 on smithi187 with /dev/vg_nvme/lv_1... 2023-07-14T18:09:11.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T18:09:11.400 INFO:journalctl@ceph.osd.10.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:09:11.931 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:09:12.342 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:11 smithi187 ceph-mon[129317]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:12.343 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.343 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.343 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.344 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:11 smithi115 ceph-mon[127451]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:12.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:11 smithi150 ceph-mon[128853]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:12.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:12.732 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:09:12 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:09:12.415+0000 7f6840b7e640 -1 osd.10 0 log_to_monitors true 2023-07-14T18:09:13.639 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T18:09:13.640 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T18:09:13.640 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2023-07-14T18:09:13.640 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2023-07-14T18:09:13.640 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128349 s, 817 MB/s 2023-07-14T18:09:13.641 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2023-07-14T18:09:13.715 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:13 smithi150 ceph-mon[128853]: from='osd.10 [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:09:13.715 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:13 smithi150 ceph-mon[128853]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:09:13.716 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:13 smithi150 ceph-mon[128853]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:13.774 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:13 smithi187 ceph-mon[129317]: from='osd.10 [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:09:13.774 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:13 smithi187 ceph-mon[129317]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:09:13.775 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:13 smithi187 ceph-mon[129317]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:13.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:13 smithi115 ceph-mon[127451]: from='osd.10 [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:09:13.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:13 smithi115 ceph-mon[127451]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:09:13.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:13 smithi115 ceph-mon[127451]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:14.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi187:vg_nvme/lv_1 2023-07-14T18:09:14.631 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:09:14.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T18:09:14.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: osdmap e69: 11 total, 10 up, 11 in 2023-07-14T18:09:14.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:14.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='osd.10 [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:09:14.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:09:14.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi150 2023-07-14T18:09:14.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi150 to 4292M 2023-07-14T18:09:14.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:14.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:14.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:14.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:14.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:14.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.869 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:09:14.409+0000 7f68361f6700 -1 osd.10 0 waiting for initial osdmap 2023-07-14T18:09:14.869 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:09:14.421+0000 7f6831fec700 -1 osd.10 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:09:14.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T18:09:14.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: osdmap e69: 11 total, 10 up, 11 in 2023-07-14T18:09:14.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:14.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='osd.10 [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:09:14.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:09:14.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi150 2023-07-14T18:09:14.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi150 to 4292M 2023-07-14T18:09:14.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:14.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:14.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:14.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:14.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:14.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.880 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T18:09:14.880 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: osdmap e69: 11 total, 10 up, 11 in 2023-07-14T18:09:14.880 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:14.880 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='osd.10 [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:09:14.881 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-07-14T18:09:14.881 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi150 2023-07-14T18:09:14.881 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.881 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.881 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.881 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.881 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.881 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:14.882 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi150 to 4292M 2023-07-14T18:09:14.882 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.882 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:14.882 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:14.882 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:14.882 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:14.882 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:14.883 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:14.883 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:15.714 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:15 smithi187 ceph-mon[129317]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:09:15.715 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:15 smithi187 ceph-mon[129317]: osdmap e70: 11 total, 10 up, 11 in 2023-07-14T18:09:15.715 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:15.715 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:15.715 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:15 smithi187 ceph-mon[129317]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:15.846 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 -- 172.21.15.187:0/1675256048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801030c0 msgr2=0x7fde801054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:15.846 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 --2- 172.21.15.187:0/1675256048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801030c0 0x7fde801054a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fde70009990 tx=0x7fde70009ca0 comp rx=0 tx=0).stop 2023-07-14T18:09:15.846 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 -- 172.21.15.187:0/1675256048 shutdown_connections 2023-07-14T18:09:15.846 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 --2- 172.21.15.187:0/1675256048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde801059e0 0x7fde80107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:15.847 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 --2- 172.21.15.187:0/1675256048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde801007a0 0x7fde80102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:15.847 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 --2- 172.21.15.187:0/1675256048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801030c0 0x7fde801054a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:15.847 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 -- 172.21.15.187:0/1675256048 >> 172.21.15.187:0/1675256048 conn(0x7fde800fa1e0 msgr2=0x7fde800fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:15.847 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 -- 172.21.15.187:0/1675256048 shutdown_connections 2023-07-14T18:09:15.847 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 -- 172.21.15.187:0/1675256048 wait complete. 2023-07-14T18:09:15.847 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.808+0000 7fde85691700 1 Processor -- start 2023-07-14T18:09:15.849 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde85691700 1 -- start start 2023-07-14T18:09:15.849 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde85691700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde801007a0 0x7fde80195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:15.849 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde85691700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde801030c0 0x7fde80195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:15.849 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde85691700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801059e0 0x7fde8019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:15.849 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde85691700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8019d400 con 0x7fde801030c0 2023-07-14T18:09:15.849 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde85691700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8019d540 con 0x7fde801007a0 2023-07-14T18:09:15.850 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde85691700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8019d6d0 con 0x7fde801059e0 2023-07-14T18:09:15.850 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801059e0 0x7fde8019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:15.850 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7f7fe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801059e0 0x7fde8019ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:58814/0 (socket says 172.21.15.187:58814) 2023-07-14T18:09:15.850 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7f7fe700 1 -- 172.21.15.187:0/804417561 learned_addr learned my addr 172.21.15.187:0/804417561 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:09:15.850 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7e7fc700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde801030c0 0x7fde80195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:15.850 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7effd700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde801007a0 0x7fde80195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:15.850 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7f7fe700 1 -- 172.21.15.187:0/804417561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde801007a0 msgr2=0x7fde80195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:15.851 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7f7fe700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde801007a0 0x7fde80195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:15.851 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7f7fe700 1 -- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde801030c0 msgr2=0x7fde80195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:15.851 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7f7fe700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde801030c0 0x7fde80195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:15.851 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7f7fe700 1 -- 172.21.15.187:0/804417561 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700095f0 con 0x7fde801059e0 2023-07-14T18:09:15.851 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.809+0000 7fde7effd700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde801007a0 0x7fde80195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:09:15.851 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.810+0000 7fde7f7fe700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801059e0 0x7fde8019ce30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde78007d10 tx=0x7fde7800a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:15.851 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.810+0000 7fde67fff700 1 -- 172.21.15.187:0/804417561 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7800ad60 con 0x7fde801059e0 2023-07-14T18:09:15.851 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.810+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8019d9b0 con 0x7fde801059e0 2023-07-14T18:09:15.852 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.810+0000 7fde67fff700 1 -- 172.21.15.187:0/804417561 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fde78007e60 con 0x7fde801059e0 2023-07-14T18:09:15.852 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.810+0000 7fde67fff700 1 -- 172.21.15.187:0/804417561 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde780115e0 con 0x7fde801059e0 2023-07-14T18:09:15.852 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.810+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8019de60 con 0x7fde801059e0 2023-07-14T18:09:15.852 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.811+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde800fbd50 con 0x7fde801059e0 2023-07-14T18:09:15.853 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.812+0000 7fde67fff700 1 -- 172.21.15.187:0/804417561 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fde78012940 con 0x7fde801059e0 2023-07-14T18:09:15.853 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.812+0000 7fde67fff700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fde680a1150 0x7fde680a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:15.853 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.812+0000 7fde67fff700 1 -- 172.21.15.187:0/804417561 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6727+0+0 (secure 0 0 0) 0x7fde780cc530 con 0x7fde801059e0 2023-07-14T18:09:15.853 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.813+0000 7fde7effd700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fde680a1150 0x7fde680a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:15.853 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.813+0000 7fde7effd700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fde680a1150 0x7fde680a3600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde74005fd0 tx=0x7fde74005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:15.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:15 smithi115 ceph-mon[127451]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:09:15.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:15 smithi115 ceph-mon[127451]: osdmap e70: 11 total, 10 up, 11 in 2023-07-14T18:09:15.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:15.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:15.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:15 smithi115 ceph-mon[127451]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:15.855 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:15.817+0000 7fde67fff700 1 -- 172.21.15.187:0/804417561 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fde780962b0 con 0x7fde801059e0 2023-07-14T18:09:15.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:15 smithi150 ceph-mon[128853]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-07-14T18:09:15.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:15 smithi150 ceph-mon[128853]: osdmap e70: 11 total, 10 up, 11 in 2023-07-14T18:09:15.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:15.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:15.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:15 smithi150 ceph-mon[128853]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:09:16.048 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:16.010+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fde80060a80 con 0x7fde680a1150 2023-07-14T18:09:16.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:16 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:09:16.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:16 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:09:16.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:16 smithi115 ceph-mon[127451]: osd.10 [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] boot 2023-07-14T18:09:16.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:16 smithi115 ceph-mon[127451]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T18:09:16.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:16 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:16.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:16 smithi115 ceph-mon[127451]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:09:16.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:16 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:09:16.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:16 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:09:16.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:16 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:16.863 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:16 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:09:16.863 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:16 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:09:16.863 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:16 smithi187 ceph-mon[129317]: osd.10 [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] boot 2023-07-14T18:09:16.863 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:16 smithi187 ceph-mon[129317]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T18:09:16.863 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:16 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:16.863 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:16 smithi187 ceph-mon[129317]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:09:16.864 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:16 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:09:16.864 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:16 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:09:16.864 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:16 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:16.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:16 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:09:16.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:16 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:09:16.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:16 smithi150 ceph-mon[128853]: osd.10 [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] boot 2023-07-14T18:09:16.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:16 smithi150 ceph-mon[128853]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T18:09:16.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:16 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:09:16.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:16 smithi150 ceph-mon[128853]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:09:16.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:16 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:09:16.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:16 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:09:16.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:16 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:17.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:17 smithi187 ceph-mon[129317]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T18:09:17.764 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:17 smithi187 ceph-mon[129317]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:17.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:17 smithi115 ceph-mon[127451]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T18:09:17.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:17 smithi115 ceph-mon[127451]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:17.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:17 smithi150 ceph-mon[128853]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T18:09:17.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:17 smithi150 ceph-mon[128853]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:18.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:18 smithi115 ceph-mon[127451]: osdmap e73: 11 total, 11 up, 11 in 2023-07-14T18:09:18.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:18 smithi150 ceph-mon[128853]: osdmap e73: 11 total, 11 up, 11 in 2023-07-14T18:09:18.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:18 smithi187 ceph-mon[129317]: osdmap e73: 11 total, 11 up, 11 in 2023-07-14T18:09:19.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:19 smithi115 ceph-mon[127451]: osdmap e74: 11 total, 11 up, 11 in 2023-07-14T18:09:19.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:19 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/3535456129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "646460b3-b30a-4570-8b1a-dce626e03f2c"}]: dispatch 2023-07-14T18:09:19.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:19 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "646460b3-b30a-4570-8b1a-dce626e03f2c"}]: dispatch 2023-07-14T18:09:19.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:19 smithi115 ceph-mon[127451]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "646460b3-b30a-4570-8b1a-dce626e03f2c"}]': finished 2023-07-14T18:09:19.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:19 smithi115 ceph-mon[127451]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T18:09:19.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:19.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:19 smithi115 ceph-mon[127451]: pgmap v210: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 167 KiB/s, 0 objects/s recovering 2023-07-14T18:09:19.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:19 smithi115 ceph-mon[127451]: from='client.? 172.21.15.187:0/3154604252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:09:19.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:19 smithi150 ceph-mon[128853]: osdmap e74: 11 total, 11 up, 11 in 2023-07-14T18:09:19.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:19 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/3535456129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "646460b3-b30a-4570-8b1a-dce626e03f2c"}]: dispatch 2023-07-14T18:09:19.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:19 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "646460b3-b30a-4570-8b1a-dce626e03f2c"}]: dispatch 2023-07-14T18:09:19.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:19 smithi150 ceph-mon[128853]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "646460b3-b30a-4570-8b1a-dce626e03f2c"}]': finished 2023-07-14T18:09:19.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:19 smithi150 ceph-mon[128853]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T18:09:19.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:19.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:19 smithi150 ceph-mon[128853]: pgmap v210: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 167 KiB/s, 0 objects/s recovering 2023-07-14T18:09:19.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:19 smithi150 ceph-mon[128853]: from='client.? 172.21.15.187:0/3154604252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:09:19.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:19 smithi187 ceph-mon[129317]: osdmap e74: 11 total, 11 up, 11 in 2023-07-14T18:09:19.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:19 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/3535456129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "646460b3-b30a-4570-8b1a-dce626e03f2c"}]: dispatch 2023-07-14T18:09:19.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:19 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "646460b3-b30a-4570-8b1a-dce626e03f2c"}]: dispatch 2023-07-14T18:09:19.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:19 smithi187 ceph-mon[129317]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "646460b3-b30a-4570-8b1a-dce626e03f2c"}]': finished 2023-07-14T18:09:19.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:19 smithi187 ceph-mon[129317]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T18:09:19.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:19.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:19 smithi187 ceph-mon[129317]: pgmap v210: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 167 KiB/s, 0 objects/s recovering 2023-07-14T18:09:19.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:19 smithi187 ceph-mon[129317]: from='client.? 172.21.15.187:0/3154604252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:09:22.226 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:21 smithi187 ceph-mon[129317]: pgmap v211: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2023-07-14T18:09:22.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:21 smithi115 ceph-mon[127451]: pgmap v211: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2023-07-14T18:09:22.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:21 smithi150 ceph-mon[128853]: pgmap v211: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 130 KiB/s, 0 objects/s recovering 2023-07-14T18:09:23.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:23 smithi187 ceph-mon[129317]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:09:23.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:23 smithi115 ceph-mon[127451]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:09:23.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:23 smithi150 ceph-mon[128853]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:09:26.097 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:25 smithi187 ceph-mon[129317]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-14T18:09:26.097 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T18:09:26.097 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:25 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:26.097 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:25 smithi187 ceph-mon[129317]: Deploying daemon osd.11 on smithi187 2023-07-14T18:09:26.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:25 smithi115 ceph-mon[127451]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-14T18:09:26.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T18:09:26.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:25 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:26.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:25 smithi115 ceph-mon[127451]: Deploying daemon osd.11 on smithi187 2023-07-14T18:09:26.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:25 smithi150 ceph-mon[128853]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-14T18:09:26.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T18:09:26.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:25 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:26.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:25 smithi150 ceph-mon[128853]: Deploying daemon osd.11 on smithi187 2023-07-14T18:09:28.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:27 smithi187 ceph-mon[129317]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:28.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:27 smithi115 ceph-mon[127451]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:28.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:27 smithi150 ceph-mon[128853]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:30.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:29 smithi187 ceph-mon[129317]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:30.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:30.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:30.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:30.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:29 smithi115 ceph-mon[127451]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:30.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:30.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:30.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:30.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:29 smithi150 ceph-mon[128853]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:30.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:30.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:30.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:32.229 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:31 smithi187 ceph-mon[129317]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:32.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:31 smithi115 ceph-mon[127451]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:32.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:31 smithi150 ceph-mon[128853]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:32.606 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.569+0000 7fde67fff700 1 -- 172.21.15.187:0/804417561 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fde80060a80 con 0x7fde680a1150 2023-07-14T18:09:32.607 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 11 on host 'smithi187' 2023-07-14T18:09:32.609 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fde680a1150 msgr2=0x7fde680a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:32.609 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fde680a1150 0x7fde680a3600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde74005fd0 tx=0x7fde74005ee0 comp rx=0 tx=0).stop 2023-07-14T18:09:32.609 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801059e0 msgr2=0x7fde8019ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:32.610 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801059e0 0x7fde8019ce30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde78007d10 tx=0x7fde7800a340 comp rx=0 tx=0).stop 2023-07-14T18:09:32.610 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 shutdown_connections 2023-07-14T18:09:32.610 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fde680a1150 0x7fde680a3600 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:32.610 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fde801059e0 0x7fde8019ce30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:32.611 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde801007a0 0x7fde80195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:32.611 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 --2- 172.21.15.187:0/804417561 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fde801030c0 0x7fde80195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:32.611 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 >> 172.21.15.187:0/804417561 conn(0x7fde800fa1e0 msgr2=0x7fde80101420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:32.611 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 shutdown_connections 2023-07-14T18:09:32.612 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:32.571+0000 7fde85691700 1 -- 172.21.15.187:0/804417561 wait complete. 2023-07-14T18:09:33.035 DEBUG:teuthology.orchestra.run.smithi187:osd.11> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.11.service 2023-07-14T18:09:33.037 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-14T18:09:33.038 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T18:09:33.078 INFO:journalctl@ceph.osd.11.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:09:33.605 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:09:33.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.832 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:33 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.833 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:33 smithi115 ceph-mon[127451]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:33.869 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:33 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:33 smithi150 ceph-mon[128853]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:33.888 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.888 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.888 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.888 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:33 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:33.889 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:33 smithi187 ceph-mon[129317]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:33.889 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:09:33 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:09:33.749+0000 7fef31a7c640 -1 osd.11 0 log_to_monitors true 2023-07-14T18:09:34.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:34 smithi187 ceph-mon[129317]: from='osd.11 [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:09:34.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:34 smithi187 ceph-mon[129317]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:09:34.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.856+0000 7f447f70e700 1 -- 172.21.15.115:0/3101314040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 msgr2=0x7f44781070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:34.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.856+0000 7f447f70e700 1 --2- 172.21.15.115:0/3101314040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 0x7f44781070c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f447000b0a0 tx=0x7f447000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:09:34.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.856+0000 7f447f70e700 1 -- 172.21.15.115:0/3101314040 shutdown_connections 2023-07-14T18:09:34.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.856+0000 7f447f70e700 1 --2- 172.21.15.115:0/3101314040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 0x7f44781070c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:34.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.856+0000 7f447f70e700 1 --2- 172.21.15.115:0/3101314040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44780febd0 0x7f44780fefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:34.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.856+0000 7f447f70e700 1 --2- 172.21.15.115:0/3101314040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44780ffe70 0x7f44781002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:34.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.856+0000 7f447f70e700 1 -- 172.21.15.115:0/3101314040 >> 172.21.15.115:0/3101314040 conn(0x7f44780fa240 msgr2=0x7f44780fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:34.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.856+0000 7f447f70e700 1 -- 172.21.15.115:0/3101314040 shutdown_connections 2023-07-14T18:09:34.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.856+0000 7f447f70e700 1 -- 172.21.15.115:0/3101314040 wait complete. 2023-07-14T18:09:34.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447f70e700 1 Processor -- start 2023-07-14T18:09:34.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447f70e700 1 -- start start 2023-07-14T18:09:34.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447f70e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44780febd0 0x7f44780695a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:34.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447f70e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44780ffe70 0x7f447819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:34.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447f70e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 0x7f447819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:34.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447f70e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447819d570 con 0x7f4478100820 2023-07-14T18:09:34.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447f70e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447819d6e0 con 0x7f44780ffe70 2023-07-14T18:09:34.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447f70e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447819d8a0 con 0x7f44780febd0 2023-07-14T18:09:34.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447dcab700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 0x7f447819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:34.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447dcab700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 0x7f447819cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52154/0 (socket says 172.21.15.115:52154) 2023-07-14T18:09:34.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447d4aa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44780febd0 0x7f44780695a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:34.897 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447dcab700 1 -- 172.21.15.115:0/4055626454 learned_addr learned my addr 172.21.15.115:0/4055626454 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:34.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447cca9700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44780ffe70 0x7f447819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:34.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447dcab700 1 -- 172.21.15.115:0/4055626454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44780febd0 msgr2=0x7f44780695a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:34.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447dcab700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44780febd0 0x7f44780695a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:34.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447dcab700 1 -- 172.21.15.115:0/4055626454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44780ffe70 msgr2=0x7f447819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:34.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447dcab700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44780ffe70 0x7f447819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:34.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447dcab700 1 -- 172.21.15.115:0/4055626454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4470009d00 con 0x7f4478100820 2023-07-14T18:09:34.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447cca9700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44780ffe70 0x7f447819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:09:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.857+0000 7f447dcab700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 0x7f447819cdd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f447000b780 tx=0x7f447000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.858+0000 7f446e7fc700 1 -- 172.21.15.115:0/4055626454 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447000e040 con 0x7f4478100820 2023-07-14T18:09:34.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.858+0000 7f446e7fc700 1 -- 172.21.15.115:0/4055626454 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4470004e60 con 0x7f4478100820 2023-07-14T18:09:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.858+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447819db20 con 0x7f4478100820 2023-07-14T18:09:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.858+0000 7f446e7fc700 1 -- 172.21.15.115:0/4055626454 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44700052a0 con 0x7f4478100820 2023-07-14T18:09:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.858+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447819e060 con 0x7f4478100820 2023-07-14T18:09:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.859+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44781062c0 con 0x7f4478100820 2023-07-14T18:09:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.860+0000 7f446e7fc700 1 -- 172.21.15.115:0/4055626454 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f447001d470 con 0x7f4478100820 2023-07-14T18:09:34.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.860+0000 7f446e7fc700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f44640a11d0 0x7f44640a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:34.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.860+0000 7f447d4aa700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f44640a11d0 0x7f44640a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:34.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.860+0000 7f446e7fc700 1 -- 172.21.15.115:0/4055626454 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f44700cd980 con 0x7f4478100820 2023-07-14T18:09:34.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.861+0000 7f447d4aa700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f44640a11d0 0x7f44640a3680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4468009a10 tx=0x7f4468009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:34.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:34.864+0000 7f446e7fc700 1 -- 172.21.15.115:0/4055626454 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4470097730 con 0x7f4478100820 2023-07-14T18:09:35.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.050+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f447802c8e0 con 0x7f4478100820 2023-07-14T18:09:35.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.051+0000 7f446e7fc700 1 -- 172.21.15.115:0/4055626454 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7f447002d2f0 con 0x7f4478100820 2023-07-14T18:09:35.089 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:35.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f44640a11d0 msgr2=0x7f44640a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:35.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f44640a11d0 0x7f44640a3680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4468009a10 tx=0x7f4468009190 comp rx=0 tx=0).stop 2023-07-14T18:09:35.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 msgr2=0x7f447819cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 0x7f447819cdd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f447000b780 tx=0x7f447000bd40 comp rx=0 tx=0).stop 2023-07-14T18:09:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 shutdown_connections 2023-07-14T18:09:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f44640a11d0 0x7f44640a3680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4478100820 0x7f447819cdd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f44780febd0 0x7f44780695a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 --2- 172.21.15.115:0/4055626454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44780ffe70 0x7f447819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 >> 172.21.15.115:0/4055626454 conn(0x7f44780fa240 msgr2=0x7f4478103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:35.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 shutdown_connections 2023-07-14T18:09:35.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:35.053+0000 7f447f70e700 1 -- 172.21.15.115:0/4055626454 wait complete. 2023-07-14T18:09:35.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:34 smithi115 ceph-mon[127451]: from='osd.11 [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:09:35.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:34 smithi115 ceph-mon[127451]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:09:35.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:34 smithi150 ceph-mon[128853]: from='osd.11 [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:09:35.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:34 smithi150 ceph-mon[128853]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:09:35.448 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":76,"num_osds":12,"num_up_osds":11,"osd_up_since":1689358155,"num_in_osds":12,"osd_in_since":1689358158,"num_remapped_pgs":0} 2023-07-14T18:09:35.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T18:09:35.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: osdmap e76: 12 total, 11 up, 12 in 2023-07-14T18:09:35.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:35.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='osd.11 [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:09:35.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:09:35.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:35.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4055626454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:09:35.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: Detected new or changed devices on smithi187 2023-07-14T18:09:35.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:35.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:35.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:35.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:35.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:35.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:35.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: Adjusting osd_memory_target on smithi187 to 4292M 2023-07-14T18:09:35.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:35.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:35.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:35.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:35.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:35.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:35.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:35.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:35 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:35.932 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:09:35.620+0000 7fef270f4700 -1 osd.11 0 waiting for initial osdmap 2023-07-14T18:09:35.933 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:09:35.631+0000 7fef1eee2700 -1 osd.11 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:09:35.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T18:09:35.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: osdmap e76: 12 total, 11 up, 12 in 2023-07-14T18:09:35.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:35.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='osd.11 [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:09:35.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:09:35.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:35.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/4055626454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:09:35.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: Detected new or changed devices on smithi187 2023-07-14T18:09:35.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:35.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:35.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:35.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:35.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:35.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:35.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: Adjusting osd_memory_target on smithi187 to 4292M 2023-07-14T18:09:35.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:35.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:35.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:35.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:35.936 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:35.936 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:35.936 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:35.936 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:35 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:36.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T18:09:36.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: osdmap e76: 12 total, 11 up, 12 in 2023-07-14T18:09:36.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:36.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='osd.11 [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:09:36.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-07-14T18:09:36.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:36.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/4055626454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:09:36.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: Detected new or changed devices on smithi187 2023-07-14T18:09:36.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:36.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:36.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:36.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:36.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:36.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:09:36.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: Adjusting osd_memory_target on smithi187 to 4292M 2023-07-14T18:09:36.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:36.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:36.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:36.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:36.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:36.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:36.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:36.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:35 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:36.449 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T18:09:36.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:36 smithi115 ceph-mon[127451]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:09:36.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:36 smithi115 ceph-mon[127451]: osdmap e77: 12 total, 11 up, 12 in 2023-07-14T18:09:36.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:36 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:36.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:36 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:36.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:36 smithi187 ceph-mon[129317]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:09:36.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:36 smithi187 ceph-mon[129317]: osdmap e77: 12 total, 11 up, 12 in 2023-07-14T18:09:36.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:36 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:36.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:36 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:37.000 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:09:37.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:36 smithi150 ceph-mon[128853]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-07-14T18:09:37.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:36 smithi150 ceph-mon[128853]: osdmap e77: 12 total, 11 up, 12 in 2023-07-14T18:09:37.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:36 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:37.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:36 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:37.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:37 smithi115 ceph-mon[127451]: purged_snaps scrub starts 2023-07-14T18:09:37.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:37 smithi115 ceph-mon[127451]: purged_snaps scrub ok 2023-07-14T18:09:37.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:37 smithi115 ceph-mon[127451]: osd.11 [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] boot 2023-07-14T18:09:37.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:37 smithi115 ceph-mon[127451]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T18:09:37.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:37 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:37.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:37 smithi115 ceph-mon[127451]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:37.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:37 smithi187 ceph-mon[129317]: purged_snaps scrub starts 2023-07-14T18:09:37.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:37 smithi187 ceph-mon[129317]: purged_snaps scrub ok 2023-07-14T18:09:37.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:37 smithi187 ceph-mon[129317]: osd.11 [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] boot 2023-07-14T18:09:37.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:37 smithi187 ceph-mon[129317]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T18:09:37.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:37 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:37.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:37 smithi187 ceph-mon[129317]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:38.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:37 smithi150 ceph-mon[128853]: purged_snaps scrub starts 2023-07-14T18:09:38.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:37 smithi150 ceph-mon[128853]: purged_snaps scrub ok 2023-07-14T18:09:38.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:37 smithi150 ceph-mon[128853]: osd.11 [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] boot 2023-07-14T18:09:38.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:37 smithi150 ceph-mon[128853]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T18:09:38.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:37 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:09:38.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:37 smithi150 ceph-mon[128853]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:09:38.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.147+0000 7f7abbee0700 1 -- 172.21.15.115:0/290282358 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab4100740 msgr2=0x7f7ab4107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:38.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.147+0000 7f7abbee0700 1 --2- 172.21.15.115:0/290282358 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab4100740 0x7f7ab4107000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00b0a0 tx=0x7f7aac00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.147+0000 7f7abbee0700 1 -- 172.21.15.115:0/290282358 shutdown_connections 2023-07-14T18:09:38.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.147+0000 7f7abbee0700 1 --2- 172.21.15.115:0/290282358 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab4100740 0x7f7ab4107000 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.147+0000 7f7abbee0700 1 --2- 172.21.15.115:0/290282358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ab40febb0 0x7f7ab40fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.147+0000 7f7abbee0700 1 --2- 172.21.15.115:0/290282358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab40ffdb0 0x7f7ab4100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.147+0000 7f7abbee0700 1 -- 172.21.15.115:0/290282358 >> 172.21.15.115:0/290282358 conn(0x7f7ab40fa1e0 msgr2=0x7f7ab40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:38.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.147+0000 7f7abbee0700 1 -- 172.21.15.115:0/290282358 shutdown_connections 2023-07-14T18:09:38.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.147+0000 7f7abbee0700 1 -- 172.21.15.115:0/290282358 wait complete. 2023-07-14T18:09:38.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7abbee0700 1 Processor -- start 2023-07-14T18:09:38.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7abbee0700 1 -- start start 2023-07-14T18:09:38.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7abbee0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab40febb0 0x7f7ab41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:38.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7abbee0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab40ffdb0 0x7f7ab4195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:38.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7abbee0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ab4100740 0x7f7ab419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:38.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7abbee0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab419d530 con 0x7f7ab40febb0 2023-07-14T18:09:38.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7abbee0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab419d670 con 0x7f7ab40ffdb0 2023-07-14T18:09:38.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7abbee0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab419d800 con 0x7f7ab4100740 2023-07-14T18:09:38.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7ab9c7c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab40febb0 0x7f7ab41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:38.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7ab9c7c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab40febb0 0x7f7ab41959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52174/0 (socket says 172.21.15.115:52174) 2023-07-14T18:09:38.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7ab9c7c700 1 -- 172.21.15.115:0/1811780595 learned_addr learned my addr 172.21.15.115:0/1811780595 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:38.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.148+0000 7f7ab947b700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab40ffdb0 0x7f7ab4195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:38.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7aba47d700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ab4100740 0x7f7ab419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:38.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7ab9c7c700 1 -- 172.21.15.115:0/1811780595 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ab4100740 msgr2=0x7f7ab419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:38.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7ab9c7c700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ab4100740 0x7f7ab419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7ab9c7c700 1 -- 172.21.15.115:0/1811780595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab40ffdb0 msgr2=0x7f7ab4195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:38.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7ab9c7c700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab40ffdb0 0x7f7ab4195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7ab9c7c700 1 -- 172.21.15.115:0/1811780595 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac009d00 con 0x7f7ab40febb0 2023-07-14T18:09:38.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7aba47d700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ab4100740 0x7f7ab419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:09:38.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7ab9c7c700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab40febb0 0x7f7ab41959a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000c9a0 tx=0x7f7ab000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:38.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7aa6ffd700 1 -- 172.21.15.115:0/1811780595 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0004220 con 0x7f7ab40febb0 2023-07-14T18:09:38.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7aa6ffd700 1 -- 172.21.15.115:0/1811780595 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7ab0007cf0 con 0x7f7ab40febb0 2023-07-14T18:09:38.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab419dae0 con 0x7f7ab40febb0 2023-07-14T18:09:38.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7aa6ffd700 1 -- 172.21.15.115:0/1811780595 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0003b30 con 0x7f7ab40febb0 2023-07-14T18:09:38.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.149+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab419dfe0 con 0x7f7ab40febb0 2023-07-14T18:09:38.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.150+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab410a280 con 0x7f7ab40febb0 2023-07-14T18:09:38.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.151+0000 7f7aa6ffd700 1 -- 172.21.15.115:0/1811780595 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7ab0003cb0 con 0x7f7ab40febb0 2023-07-14T18:09:38.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.152+0000 7f7aa6ffd700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7aa00a1150 0x7f7aa00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:38.192 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.152+0000 7f7ab947b700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7aa00a1150 0x7f7aa00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:38.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.152+0000 7f7aa6ffd700 1 -- 172.21.15.115:0/1811780595 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7ab00cc4f0 con 0x7f7ab40febb0 2023-07-14T18:09:38.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.152+0000 7f7ab947b700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7aa00a1150 0x7f7aa00a3600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800ac50 tx=0x7f7aa800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:38.193 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.155+0000 7f7aa6ffd700 1 -- 172.21.15.115:0/1811780595 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7ab0096040 con 0x7f7ab40febb0 2023-07-14T18:09:38.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.344+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f7ab419e6b0 con 0x7f7ab40febb0 2023-07-14T18:09:38.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.344+0000 7f7aa6ffd700 1 -- 172.21.15.115:0/1811780595 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7f7ab0099660 con 0x7f7ab40febb0 2023-07-14T18:09:38.382 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:38.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.346+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7aa00a1150 msgr2=0x7f7aa00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:38.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.346+0000 7f7abbee0700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7aa00a1150 0x7f7aa00a3600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800ac50 tx=0x7f7aa800a380 comp rx=0 tx=0).stop 2023-07-14T18:09:38.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.346+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab40febb0 msgr2=0x7f7ab41959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:38.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.346+0000 7f7abbee0700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab40febb0 0x7f7ab41959a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000c9a0 tx=0x7f7ab000ccb0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.347+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 shutdown_connections 2023-07-14T18:09:38.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.347+0000 7f7abbee0700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7aa00a1150 0x7f7aa00a3600 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.347+0000 7f7abbee0700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ab4100740 0x7f7ab419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.347+0000 7f7abbee0700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7ab40febb0 0x7f7ab41959a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.347+0000 7f7abbee0700 1 --2- 172.21.15.115:0/1811780595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ab40ffdb0 0x7f7ab4195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:38.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.347+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 >> 172.21.15.115:0/1811780595 conn(0x7f7ab40fa1e0 msgr2=0x7f7ab4068140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:38.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.347+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 shutdown_connections 2023-07-14T18:09:38.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:38.347+0000 7f7abbee0700 1 -- 172.21.15.115:0/1811780595 wait complete. 2023-07-14T18:09:38.743 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":79,"num_osds":12,"num_up_osds":12,"osd_up_since":1689358176,"num_in_osds":12,"osd_in_since":1689358158,"num_remapped_pgs":0} 2023-07-14T18:09:38.744 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T18:09:38.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:38 smithi187 ceph-mon[129317]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T18:09:38.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:38 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1811780595' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:09:39.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:38 smithi115 ceph-mon[127451]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T18:09:39.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:38 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1811780595' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:09:39.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:38 smithi150 ceph-mon[128853]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T18:09:39.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:38 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1811780595' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:09:39.295 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:09:39.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:39 smithi115 ceph-mon[127451]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:39.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:39 smithi187 ceph-mon[129317]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:40.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:39 smithi150 ceph-mon[128853]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:40.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 -- 172.21.15.115:0/3996121890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1174104a80 msgr2=0x7f117410b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:40.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 --2- 172.21.15.115:0/3996121890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1174104a80 0x7f117410b270 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f116c00b0d0 tx=0x7f116c00b3e0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 -- 172.21.15.115:0/3996121890 shutdown_connections 2023-07-14T18:09:40.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 --2- 172.21.15.115:0/3996121890 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1174104a80 0x7f117410b270 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 --2- 172.21.15.115:0/3996121890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1174103370 0x7f1174103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 --2- 172.21.15.115:0/3996121890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11741040d0 0x7f1174104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 -- 172.21.15.115:0/3996121890 >> 172.21.15.115:0/3996121890 conn(0x7f1174074550 msgr2=0x7f1174074950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:40.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 -- 172.21.15.115:0/3996121890 shutdown_connections 2023-07-14T18:09:40.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 -- 172.21.15.115:0/3996121890 wait complete. 2023-07-14T18:09:40.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.572+0000 7f1179837700 1 Processor -- start 2023-07-14T18:09:40.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f1179837700 1 -- start start 2023-07-14T18:09:40.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f1179837700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174103370 0x7f117419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:40.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f1179837700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11741040d0 0x7f117419af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:40.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f1179837700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1174104a80 0x7f11741949f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:40.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f1179837700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11741951a0 con 0x7f11741040d0 2023-07-14T18:09:40.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f1179837700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174195310 con 0x7f1174103370 2023-07-14T18:09:40.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f1179837700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11741954d0 con 0x7f1174104a80 2023-07-14T18:09:40.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11727fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11741040d0 0x7f117419af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:40.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11727fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11741040d0 0x7f117419af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52190/0 (socket says 172.21.15.115:52190) 2023-07-14T18:09:40.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11727fc700 1 -- 172.21.15.115:0/384898982 learned_addr learned my addr 172.21.15.115:0/384898982 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:40.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f1172ffd700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174103370 0x7f117419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:40.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11737fe700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1174104a80 0x7f11741949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:40.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11727fc700 1 -- 172.21.15.115:0/384898982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1174104a80 msgr2=0x7f11741949f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:40.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11727fc700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1174104a80 0x7f11741949f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11727fc700 1 -- 172.21.15.115:0/384898982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174103370 msgr2=0x7f117419aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:40.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11727fc700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174103370 0x7f117419aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11727fc700 1 -- 172.21.15.115:0/384898982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c009d00 con 0x7f11741040d0 2023-07-14T18:09:40.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f11737fe700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1174104a80 0x7f11741949f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:09:40.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.573+0000 7f1172ffd700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174103370 0x7f117419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:09:40.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.574+0000 7f11727fc700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11741040d0 0x7f117419af40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1164007b50 tx=0x7f1164007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:40.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.574+0000 7f1178835700 1 -- 172.21.15.115:0/384898982 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1164011070 con 0x7f11741040d0 2023-07-14T18:09:40.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.574+0000 7f1178835700 1 -- 172.21.15.115:0/384898982 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f116400c2e0 con 0x7f11741040d0 2023-07-14T18:09:40.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.574+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1174195750 con 0x7f11741040d0 2023-07-14T18:09:40.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.574+0000 7f1178835700 1 -- 172.21.15.115:0/384898982 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116400a990 con 0x7f11741040d0 2023-07-14T18:09:40.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.574+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117410bd40 con 0x7f11741040d0 2023-07-14T18:09:40.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.575+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117404e340 con 0x7f11741040d0 2023-07-14T18:09:40.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.576+0000 7f1178835700 1 -- 172.21.15.115:0/384898982 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f116400aaf0 con 0x7f11741040d0 2023-07-14T18:09:40.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.576+0000 7f1178835700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f115c0a1150 0x7f115c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:40.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.576+0000 7f1172ffd700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f115c0a1150 0x7f115c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:40.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.577+0000 7f1178835700 1 -- 172.21.15.115:0/384898982 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f11640cd600 con 0x7f11741040d0 2023-07-14T18:09:40.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.577+0000 7f1172ffd700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f115c0a1150 0x7f115c0a3600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1168005950 tx=0x7f1168005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:40.617 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.580+0000 7f1178835700 1 -- 172.21.15.115:0/384898982 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f11640971f0 con 0x7f11741040d0 2023-07-14T18:09:40.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.767+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1174195f20 con 0x7f11741040d0 2023-07-14T18:09:40.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.768+0000 7f1178835700 1 -- 172.21.15.115:0/384898982 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18492 (secure 0 0 0) 0x7f116409a810 con 0x7f11741040d0 2023-07-14T18:09:40.805 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:40.806 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":79,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:09:37.624654+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:09:40.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f115c0a1150 msgr2=0x7f115c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:40.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f115c0a1150 0x7f115c0a3600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1168005950 tx=0x7f1168005f50 comp rx=0 tx=0).stop 2023-07-14T18:09:40.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11741040d0 msgr2=0x7f117419af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:40.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11741040d0 0x7f117419af40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1164007b50 tx=0x7f1164007f10 comp rx=0 tx=0).stop 2023-07-14T18:09:40.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 shutdown_connections 2023-07-14T18:09:40.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f115c0a1150 0x7f115c0a3600 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1174104a80 0x7f11741949f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174103370 0x7f117419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 --2- 172.21.15.115:0/384898982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f11741040d0 0x7f117419af40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:40.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 >> 172.21.15.115:0/384898982 conn(0x7f1174074550 msgr2=0x7f11740fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:40.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 shutdown_connections 2023-07-14T18:09:40.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:40.770+0000 7f1179837700 1 -- 172.21.15.115:0/384898982 wait complete. 2023-07-14T18:09:41.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:40 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/384898982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:09:41.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:40 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/384898982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:09:41.152 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T18:09:41.152 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-14T18:09:41.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:40 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/384898982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:09:41.705 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:09:42.119 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:41 smithi150 ceph-mon[128853]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:42.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:41 smithi115 ceph-mon[127451]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:42.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:41 smithi187 ceph-mon[129317]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:43.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.969+0000 7f0adead3700 1 -- 172.21.15.115:0/720909829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80fee40 msgr2=0x7f0ad80ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:43.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.969+0000 7f0adead3700 1 --2- 172.21.15.115:0/720909829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80fee40 0x7f0ad80ff210 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0099c0 tx=0x7f0acc009cd0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 -- 172.21.15.115:0/720909829 shutdown_connections 2023-07-14T18:09:43.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 --2- 172.21.15.115:0/720909829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad8103ee0 0x7f0ad810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 --2- 172.21.15.115:0/720909829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80fee40 0x7f0ad80ff210 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 --2- 172.21.15.115:0/720909829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad81037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 -- 172.21.15.115:0/720909829 >> 172.21.15.115:0/720909829 conn(0x7f0ad8075ca0 msgr2=0x7f0ad80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:43.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 -- 172.21.15.115:0/720909829 shutdown_connections 2023-07-14T18:09:43.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 -- 172.21.15.115:0/720909829 wait complete. 2023-07-14T18:09:43.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 Processor -- start 2023-07-14T18:09:43.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 -- start start 2023-07-14T18:09:43.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad80fee40 0x7f0ad8196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:43.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad8197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:43.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad8103ee0 0x7f0ad81a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:43.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad81979b0 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad81a20f0 con 0x7f0ad80fee40 2023-07-14T18:09:43.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.970+0000 7f0adead3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad81a22b0 con 0x7f0ad8103ee0 2023-07-14T18:09:43.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad8197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:43.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad8197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52204/0 (socket says 172.21.15.115:52204) 2023-07-14T18:09:43.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad7fff700 1 -- 172.21.15.115:0/3848810655 learned_addr learned my addr 172.21.15.115:0/3848810655 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:43.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0add070700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad8103ee0 0x7f0ad81a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:43.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0adc86f700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad80fee40 0x7f0ad8196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:43.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad7fff700 1 -- 172.21.15.115:0/3848810655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad8103ee0 msgr2=0x7f0ad81a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:43.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad7fff700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad8103ee0 0x7f0ad81a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad7fff700 1 -- 172.21.15.115:0/3848810655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad80fee40 msgr2=0x7f0ad8196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:43.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad7fff700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad80fee40 0x7f0ad8196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad7fff700 1 -- 172.21.15.115:0/3848810655 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc0095f0 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0add070700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad8103ee0 0x7f0ad81a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:09:43.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad7fff700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad8197070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400d980 tx=0x7f0ac400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:43.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad5ffb700 1 -- 172.21.15.115:0/3848810655 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4009b40 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad5ffb700 1 -- 172.21.15.115:0/3848810655 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0ac400de60 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad81a2590 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0ad5ffb700 1 -- 172.21.15.115:0/3848810655 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac400f620 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.971+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad81a2ae0 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.972+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad804e340 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.973+0000 7f0ad5ffb700 1 -- 172.21.15.115:0/3848810655 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f0ac4010460 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.974+0000 7f0ad5ffb700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ac80a11d0 0x7f0ac80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:43.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.974+0000 7f0adc86f700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ac80a11d0 0x7f0ac80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:43.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.974+0000 7f0ad5ffb700 1 -- 172.21.15.115:0/3848810655 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0ac40cbfc0 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.974+0000 7f0adc86f700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ac80a11d0 0x7f0ac80a3680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009fd0 tx=0x7f0acc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:43.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:42.978+0000 7f0ad5ffb700 1 -- 172.21.15.115:0/3848810655 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ac40fe780 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.164+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ad81a2d00 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.164+0000 7f0ad5ffb700 1 -- 172.21.15.115:0/3848810655 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ac4095bb0 con 0x7f0ad80ff7e0 2023-07-14T18:09:43.202 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:09:43.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ac80a11d0 msgr2=0x7f0ac80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:43.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ac80a11d0 0x7f0ac80a3680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009fd0 tx=0x7f0acc005fb0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80ff7e0 msgr2=0x7f0ad8197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:43.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad8197070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400d980 tx=0x7f0ac400dd40 comp rx=0 tx=0).stop 2023-07-14T18:09:43.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 shutdown_connections 2023-07-14T18:09:43.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ac80a11d0 0x7f0ac80a3680 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ad8103ee0 0x7f0ad81a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad80fee40 0x7f0ad8196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 --2- 172.21.15.115:0/3848810655 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad80ff7e0 0x7f0ad8197070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:43.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 >> 172.21.15.115:0/3848810655 conn(0x7f0ad8075ca0 msgr2=0x7f0ad8102790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:43.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 shutdown_connections 2023-07-14T18:09:43.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:43.166+0000 7f0adead3700 1 -- 172.21.15.115:0/3848810655 wait complete. 2023-07-14T18:09:43.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:43 smithi115 ceph-mon[127451]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:43.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:43 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3848810655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:09:43.648 INFO:tasks.cephadm:Adding mds.a on smithi115 2023-07-14T18:09:43.648 INFO:tasks.cephadm:Adding mds.d on smithi115 2023-07-14T18:09:43.648 INFO:tasks.cephadm:Adding mds.g on smithi115 2023-07-14T18:09:43.649 INFO:tasks.cephadm:Adding mds.j on smithi115 2023-07-14T18:09:43.649 INFO:tasks.cephadm:Adding mds.b on smithi150 2023-07-14T18:09:43.649 INFO:tasks.cephadm:Adding mds.e on smithi150 2023-07-14T18:09:43.649 INFO:tasks.cephadm:Adding mds.h on smithi150 2023-07-14T18:09:43.649 INFO:tasks.cephadm:Adding mds.k on smithi150 2023-07-14T18:09:43.650 INFO:tasks.cephadm:Adding mds.c on smithi187 2023-07-14T18:09:43.650 INFO:tasks.cephadm:Adding mds.f on smithi187 2023-07-14T18:09:43.650 INFO:tasks.cephadm:Adding mds.i on smithi187 2023-07-14T18:09:43.650 INFO:tasks.cephadm:Adding mds.l on smithi187 2023-07-14T18:09:43.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph orch apply mds all '12;smithi115=a;smithi115=d;smithi115=g;smithi115=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l' 2023-07-14T18:09:43.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:43 smithi150 ceph-mon[128853]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:43.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:43 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3848810655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:09:43.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:43 smithi187 ceph-mon[129317]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:43.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:43 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3848810655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:09:44.219 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:09:45.461 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.422+0000 7f168d8fc700 1 -- 172.21.15.187:0/4022786036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16880fdfc0 msgr2=0x7f16880fe3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:45.462 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.422+0000 7f168d8fc700 1 --2- 172.21.15.187:0/4022786036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16880fdfc0 0x7f16880fe3d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f167c009990 tx=0x7f167c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.462 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.422+0000 7f168d8fc700 1 -- 172.21.15.187:0/4022786036 shutdown_connections 2023-07-14T18:09:45.462 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.422+0000 7f168d8fc700 1 --2- 172.21.15.187:0/4022786036 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1688101930 0x7f16881081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.462 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.422+0000 7f168d8fc700 1 --2- 172.21.15.187:0/4022786036 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16880fdfc0 0x7f16880fe3d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.462 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.422+0000 7f168d8fc700 1 --2- 172.21.15.187:0/4022786036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1688100fa0 0x7f16881013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.422+0000 7f168d8fc700 1 -- 172.21.15.187:0/4022786036 >> 172.21.15.187:0/4022786036 conn(0x7f16880f99b0 msgr2=0x7f16880fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:45.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.422+0000 7f168d8fc700 1 -- 172.21.15.187:0/4022786036 shutdown_connections 2023-07-14T18:09:45.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.422+0000 7f168d8fc700 1 -- 172.21.15.187:0/4022786036 wait complete. 2023-07-14T18:09:45.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f168d8fc700 1 Processor -- start 2023-07-14T18:09:45.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f168d8fc700 1 -- start start 2023-07-14T18:09:45.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f168d8fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16880fdfc0 0x7f1688195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:45.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f168d8fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1688100fa0 0x7f1688195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:45.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f168d8fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1688101930 0x7f168819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f168d8fc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168819d420 con 0x7f16880fdfc0 2023-07-14T18:09:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f168d8fc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168819d560 con 0x7f1688101930 2023-07-14T18:09:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f168d8fc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168819d6f0 con 0x7f1688100fa0 2023-07-14T18:09:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f16867fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1688100fa0 0x7f1688195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f16867fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1688100fa0 0x7f1688195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:58938/0 (socket says 172.21.15.187:58938) 2023-07-14T18:09:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f16867fc700 1 -- 172.21.15.187:0/1259809623 learned_addr learned my addr 172.21.15.187:0/1259809623 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-07-14T18:09:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.423+0000 7f1686ffd700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16880fdfc0 0x7f1688195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f16877fe700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1688101930 0x7f168819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f16867fc700 1 -- 172.21.15.187:0/1259809623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1688101930 msgr2=0x7f168819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f16867fc700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1688101930 0x7f168819ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f16867fc700 1 -- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16880fdfc0 msgr2=0x7f1688195890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f16867fc700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16880fdfc0 0x7f1688195890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f16867fc700 1 -- 172.21.15.187:0/1259809623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c0095f0 con 0x7f1688100fa0 2023-07-14T18:09:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f16867fc700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1688100fa0 0x7f1688195dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f167800d980 tx=0x7f167800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f168c8fa700 1 -- 172.21.15.187:0/1259809623 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678009b40 con 0x7f1688100fa0 2023-07-14T18:09:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f168c8fa700 1 -- 172.21.15.187:0/1259809623 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f167800b6e0 con 0x7f1688100fa0 2023-07-14T18:09:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168819d9d0 con 0x7f1688100fa0 2023-07-14T18:09:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f168c8fa700 1 -- 172.21.15.187:0/1259809623 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167800f3c0 con 0x7f1688100fa0 2023-07-14T18:09:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.424+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168819ded0 con 0x7f1688100fa0 2023-07-14T18:09:45.467 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.425+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1688106d30 con 0x7f1688100fa0 2023-07-14T18:09:45.467 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.426+0000 7f168c8fa700 1 -- 172.21.15.187:0/1259809623 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f1678010460 con 0x7f1688100fa0 2023-07-14T18:09:45.468 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.427+0000 7f168c8fa700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16700a1150 0x7f16700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:45.468 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.427+0000 7f168c8fa700 1 -- 172.21.15.187:0/1259809623 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f16780cc300 con 0x7f1688100fa0 2023-07-14T18:09:45.468 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.427+0000 7f1686ffd700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16700a1150 0x7f16700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:45.468 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.428+0000 7f1686ffd700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16700a1150 0x7f16700a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f167c00b5c0 tx=0x7f167c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:45.468 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.430+0000 7f168c8fa700 1 -- 172.21.15.187:0/1259809623 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1678095ef0 con 0x7f1688100fa0 2023-07-14T18:09:45.662 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.625+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi115=a;smithi115=d;smithi115=g;smithi115=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f1688060c40 con 0x7f16700a1150 2023-07-14T18:09:45.670 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.633+0000 7f168c8fa700 1 -- 172.21.15.187:0/1259809623 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f1688060c40 con 0x7f16700a1150 2023-07-14T18:09:45.671 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mds.all update... 2023-07-14T18:09:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16700a1150 msgr2=0x7f16700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16700a1150 0x7f16700a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f167c00b5c0 tx=0x7f167c005fb0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1688100fa0 msgr2=0x7f1688195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1688100fa0 0x7f1688195dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f167800d980 tx=0x7f167800dc90 comp rx=0 tx=0).stop 2023-07-14T18:09:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 shutdown_connections 2023-07-14T18:09:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16700a1150 0x7f16700a3600 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1688101930 0x7f168819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16880fdfc0 0x7f1688195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 --2- 172.21.15.187:0/1259809623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1688100fa0 0x7f1688195dd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.635+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 >> 172.21.15.187:0/1259809623 conn(0x7f16880f99b0 msgr2=0x7f16880fbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.636+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 shutdown_connections 2023-07-14T18:09:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2023-07-14T18:09:45.636+0000 7f168d8fc700 1 -- 172.21.15.187:0/1259809623 wait complete. 2023-07-14T18:09:46.079 DEBUG:teuthology.orchestra.run.smithi115:mds.a> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.a.service 2023-07-14T18:09:46.082 DEBUG:teuthology.orchestra.run.smithi115:mds.d> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.d.service 2023-07-14T18:09:46.119 INFO:journalctl@ceph.mds.a.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:09:46.120 DEBUG:teuthology.orchestra.run.smithi115:mds.g> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.g.service 2023-07-14T18:09:46.150 INFO:journalctl@ceph.mds.d.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:09:46.151 DEBUG:teuthology.orchestra.run.smithi115:mds.j> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.j.service 2023-07-14T18:09:46.178 INFO:journalctl@ceph.mds.g.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:09:46.179 DEBUG:teuthology.orchestra.run.smithi150:mds.b> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.b.service 2023-07-14T18:09:46.181 DEBUG:teuthology.orchestra.run.smithi150:mds.e> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.e.service 2023-07-14T18:09:46.205 INFO:journalctl@ceph.mds.j.smithi115.stdout:-- Logs begin at Fri 2023-07-14 17:41:32 UTC. -- 2023-07-14T18:09:46.214 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:45 smithi150 ceph-mon[128853]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:46.214 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:46.214 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:46.215 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:46.215 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:46.215 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:46.215 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:46.215 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:46.215 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:45 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:46.216 DEBUG:teuthology.orchestra.run.smithi150:mds.h> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.h.service 2023-07-14T18:09:46.219 INFO:journalctl@ceph.mds.b.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:09:46.220 DEBUG:teuthology.orchestra.run.smithi150:mds.k> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.k.service 2023-07-14T18:09:46.246 INFO:journalctl@ceph.mds.e.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:09:46.246 DEBUG:teuthology.orchestra.run.smithi187:mds.c> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.c.service 2023-07-14T18:09:46.249 DEBUG:teuthology.orchestra.run.smithi187:mds.f> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.f.service 2023-07-14T18:09:46.250 INFO:journalctl@ceph.mds.h.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:09:46.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:45 smithi187 ceph-mon[129317]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:46.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:46.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:46.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:46.273 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:46.274 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:46.274 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:46.274 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:46.274 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:45 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:46.275 INFO:journalctl@ceph.mds.k.smithi150.stdout:-- Logs begin at Fri 2023-07-14 17:41:44 UTC. -- 2023-07-14T18:09:46.275 DEBUG:teuthology.orchestra.run.smithi187:mds.i> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.i.service 2023-07-14T18:09:46.278 INFO:journalctl@ceph.mds.c.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:09:46.279 DEBUG:teuthology.orchestra.run.smithi187:mds.l> sudo journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.l.service 2023-07-14T18:09:46.300 INFO:journalctl@ceph.mds.f.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:09:46.301 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-14T18:09:46.302 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:09:46.305 INFO:journalctl@ceph.mds.i.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:09:46.326 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:45 smithi115 ceph-mon[127451]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:46.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:46.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:09:46.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:46.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:09:46.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:46.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:46.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:46.328 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:45 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:46.328 INFO:journalctl@ceph.mds.l.smithi187.stdout:-- Logs begin at Fri 2023-07-14 17:42:22 UTC. -- 2023-07-14T18:09:46.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 -- 172.21.15.115:0/171081414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe80feb00 msgr2=0x7fcbe80fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:46.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 --2- 172.21.15.115:0/171081414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe80feb00 0x7fcbe80fef10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8009990 tx=0x7fcbd8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 -- 172.21.15.115:0/171081414 shutdown_connections 2023-07-14T18:09:46.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 --2- 172.21.15.115:0/171081414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe8100680 0x7fcbe8106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 --2- 172.21.15.115:0/171081414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe80feb00 0x7fcbe80fef10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 --2- 172.21.15.115:0/171081414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe80ffcf0 0x7fcbe8100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 -- 172.21.15.115:0/171081414 >> 172.21.15.115:0/171081414 conn(0x7fcbe80fa100 msgr2=0x7fcbe80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:46.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 -- 172.21.15.115:0/171081414 shutdown_connections 2023-07-14T18:09:46.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 -- 172.21.15.115:0/171081414 wait complete. 2023-07-14T18:09:46.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 Processor -- start 2023-07-14T18:09:46.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.397+0000 7fcbee895700 1 -- start start 2023-07-14T18:09:46.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbee895700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe80feb00 0x7fcbe8194740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:46.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbee895700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe80ffcf0 0x7fcbe8194c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:46.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbee895700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe8100680 0x7fcbe819bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:46.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbee895700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe819c2c0 con 0x7fcbe8100680 2023-07-14T18:09:46.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbee895700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe819c400 con 0x7fcbe80feb00 2023-07-14T18:09:46.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbee895700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe819c590 con 0x7fcbe80ffcf0 2023-07-14T18:09:46.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbece32700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe8100680 0x7fcbe819bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:46.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbece32700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe8100680 0x7fcbe819bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52218/0 (socket says 172.21.15.115:52218) 2023-07-14T18:09:46.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbece32700 1 -- 172.21.15.115:0/3035296199 learned_addr learned my addr 172.21.15.115:0/3035296199 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:46.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbe77fe700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe80ffcf0 0x7fcbe8194c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:46.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbe7fff700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe80feb00 0x7fcbe8194740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:46.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbece32700 1 -- 172.21.15.115:0/3035296199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe80ffcf0 msgr2=0x7fcbe8194c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbece32700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe80ffcf0 0x7fcbe8194c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbece32700 1 -- 172.21.15.115:0/3035296199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe80feb00 msgr2=0x7fcbe8194740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbece32700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe80feb00 0x7fcbe8194740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbece32700 1 -- 172.21.15.115:0/3035296199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd80095f0 con 0x7fcbe8100680 2023-07-14T18:09:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbe77fe700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe80ffcf0 0x7fcbe8194c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:09:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbece32700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe8100680 0x7fcbe819bcf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0007ee0 tx=0x7fcbe000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbe57fa700 1 -- 172.21.15.115:0/3035296199 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe000ddd0 con 0x7fcbe8100680 2023-07-14T18:09:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbe57fa700 1 -- 172.21.15.115:0/3035296199 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcbe000f040 con 0x7fcbe8100680 2023-07-14T18:09:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe819c870 con 0x7fcbe8100680 2023-07-14T18:09:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbe57fa700 1 -- 172.21.15.115:0/3035296199 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0004870 con 0x7fcbe8100680 2023-07-14T18:09:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.398+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe819cd80 con 0x7fcbe8100680 2023-07-14T18:09:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.399+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe8105a80 con 0x7fcbe8100680 2023-07-14T18:09:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.400+0000 7fcbe57fa700 1 -- 172.21.15.115:0/3035296199 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fcbe0003a00 con 0x7fcbe8100680 2023-07-14T18:09:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.401+0000 7fcbe57fa700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbd00a1220 0x7fcbd00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.401+0000 7fcbe7fff700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbd00a1220 0x7fcbd00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:46.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.401+0000 7fcbe57fa700 1 -- 172.21.15.115:0/3035296199 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcbe00cc5d0 con 0x7fcbe8100680 2023-07-14T18:09:46.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.401+0000 7fcbe7fff700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbd00a1220 0x7fcbd00a36d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800b5c0 tx=0x7fcbd8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:46.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.405+0000 7fcbe57fa700 1 -- 172.21.15.115:0/3035296199 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcbe00961c0 con 0x7fcbe8100680 2023-07-14T18:09:46.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.596+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbe802cab0 con 0x7fcbe8100680 2023-07-14T18:09:46.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.597+0000 7fcbe57fa700 1 -- 172.21.15.115:0/3035296199 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18492 (secure 0 0 0) 0x7fcbe00997e0 con 0x7fcbe8100680 2023-07-14T18:09:46.635 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:46.635 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":79,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:09:37.624654+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:09:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbd00a1220 msgr2=0x7fcbd00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbd00a1220 0x7fcbd00a36d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800b5c0 tx=0x7fcbd8005fb0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe8100680 msgr2=0x7fcbe819bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe8100680 0x7fcbe819bcf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0007ee0 tx=0x7fcbe000d3b0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 shutdown_connections 2023-07-14T18:09:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbd00a1220 0x7fcbd00a36d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbe8100680 0x7fcbe819bcf0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe80feb00 0x7fcbe8194740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 --2- 172.21.15.115:0/3035296199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbe80ffcf0 0x7fcbe8194c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.599+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 >> 172.21.15.115:0/3035296199 conn(0x7fcbe80fa100 msgr2=0x7fcbe80fc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.600+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 shutdown_connections 2023-07-14T18:09:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.600+0000 7fcbee895700 1 -- 172.21.15.115:0/3035296199 wait complete. 2023-07-14T18:09:46.650 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T18:09:46.650 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:09:46.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.779+0000 7f8a29664700 1 -- 172.21.15.115:0/82372933 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a241017b0 msgr2=0x7f8a24108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:46.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.779+0000 7f8a29664700 1 --2- 172.21.15.115:0/82372933 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a241017b0 0x7f8a24108070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00b0a0 tx=0x7f8a1c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.779+0000 7f8a29664700 1 -- 172.21.15.115:0/82372933 shutdown_connections 2023-07-14T18:09:46.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.779+0000 7f8a29664700 1 --2- 172.21.15.115:0/82372933 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a241017b0 0x7f8a24108070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.779+0000 7f8a29664700 1 --2- 172.21.15.115:0/82372933 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a240ffc30 0x7f8a24100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.779+0000 7f8a29664700 1 --2- 172.21.15.115:0/82372933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a24100e20 0x7f8a24101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.779+0000 7f8a29664700 1 -- 172.21.15.115:0/82372933 >> 172.21.15.115:0/82372933 conn(0x7f8a240fb290 msgr2=0x7f8a240fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.779+0000 7f8a29664700 1 -- 172.21.15.115:0/82372933 shutdown_connections 2023-07-14T18:09:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.779+0000 7f8a29664700 1 -- 172.21.15.115:0/82372933 wait complete. 2023-07-14T18:09:46.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a29664700 1 Processor -- start 2023-07-14T18:09:46.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a29664700 1 -- start start 2023-07-14T18:09:46.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a29664700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a240ffc30 0x7f8a240743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:46.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a29664700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a24100e20 0x7f8a24072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:46.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a29664700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a241017b0 0x7f8a24072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:46.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a29664700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a24073710 con 0x7f8a240ffc30 2023-07-14T18:09:46.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a29664700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a24073880 con 0x7f8a24100e20 2023-07-14T18:09:46.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a29664700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2419fa00 con 0x7f8a241017b0 2023-07-14T18:09:46.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a22ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a240ffc30 0x7f8a240743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:46.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a22ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a240ffc30 0x7f8a240743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52228/0 (socket says 172.21.15.115:52228) 2023-07-14T18:09:46.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a22ffd700 1 -- 172.21.15.115:0/3769573160 learned_addr learned my addr 172.21.15.115:0/3769573160 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:46.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.780+0000 7f8a227fc700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a24100e20 0x7f8a24072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:46.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a237fe700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a241017b0 0x7f8a24072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:46.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a22ffd700 1 -- 172.21.15.115:0/3769573160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a241017b0 msgr2=0x7f8a24072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:46.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a22ffd700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a241017b0 0x7f8a24072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a22ffd700 1 -- 172.21.15.115:0/3769573160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a24100e20 msgr2=0x7f8a24072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:46.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a22ffd700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a24100e20 0x7f8a24072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:46.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a22ffd700 1 -- 172.21.15.115:0/3769573160 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c009d00 con 0x7f8a240ffc30 2023-07-14T18:09:46.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a237fe700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a241017b0 0x7f8a24072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:09:46.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a22ffd700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a240ffc30 0x7f8a240743f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400e970 tx=0x7f8a1400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:46.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a0bfff700 1 -- 172.21.15.115:0/3769573160 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14009790 con 0x7f8a240ffc30 2023-07-14T18:09:46.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a0bfff700 1 -- 172.21.15.115:0/3769573160 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8a1400eec0 con 0x7f8a240ffc30 2023-07-14T18:09:46.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2419fce0 con 0x7f8a240ffc30 2023-07-14T18:09:46.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a0bfff700 1 -- 172.21.15.115:0/3769573160 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14003c10 con 0x7f8a240ffc30 2023-07-14T18:09:46.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.781+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a241a01a0 con 0x7f8a240ffc30 2023-07-14T18:09:46.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.782+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a241071f0 con 0x7f8a240ffc30 2023-07-14T18:09:46.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.783+0000 7f8a0bfff700 1 -- 172.21.15.115:0/3769573160 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f8a14003d90 con 0x7f8a240ffc30 2023-07-14T18:09:46.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.783+0000 7f8a0bfff700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a0c0a1150 0x7f8a0c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:46.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.784+0000 7f8a227fc700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a0c0a1150 0x7f8a0c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:46.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.784+0000 7f8a0bfff700 1 -- 172.21.15.115:0/3769573160 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8a14014070 con 0x7f8a240ffc30 2023-07-14T18:09:46.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.784+0000 7f8a227fc700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a0c0a1150 0x7f8a0c0a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a18005950 tx=0x7f8a1800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:46.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.788+0000 7f8a0bfff700 1 -- 172.21.15.115:0/3769573160 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a14096440 con 0x7f8a240ffc30 2023-07-14T18:09:47.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.977+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a2402cb50 con 0x7f8a240ffc30 2023-07-14T18:09:47.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.978+0000 7f8a0bfff700 1 -- 172.21.15.115:0/3769573160 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a14099a60 con 0x7f8a240ffc30 2023-07-14T18:09:47.014 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:09:47.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a0c0a1150 msgr2=0x7f8a0c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:47.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a0c0a1150 0x7f8a0c0a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a18005950 tx=0x7f8a1800b410 comp rx=0 tx=0).stop 2023-07-14T18:09:47.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a240ffc30 msgr2=0x7f8a240743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:47.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a240ffc30 0x7f8a240743f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400e970 tx=0x7f8a1400ec80 comp rx=0 tx=0).stop 2023-07-14T18:09:47.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 shutdown_connections 2023-07-14T18:09:47.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a0c0a1150 0x7f8a0c0a3600 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a241017b0 0x7f8a24072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a240ffc30 0x7f8a240743f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 --2- 172.21.15.115:0/3769573160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a24100e20 0x7f8a24072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 >> 172.21.15.115:0/3769573160 conn(0x7f8a240fb290 msgr2=0x7f8a24104330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:47.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 shutdown_connections 2023-07-14T18:09:47.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:46.980+0000 7f8a29664700 1 -- 172.21.15.115:0/3769573160 wait complete. 2023-07-14T18:09:47.029 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:09:47.148 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:46 smithi187 ceph-mon[129317]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi115=a;smithi115=d;smithi115=g;smithi115=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:09:47.148 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:46 smithi187 ceph-mon[129317]: Saving service mds.all spec with placement smithi115=a;smithi115=d;smithi115=g;smithi115=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l;count:12 2023-07-14T18:09:47.148 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:46 smithi187 ceph-mon[129317]: Deploying daemon mds.l on smithi187 2023-07-14T18:09:47.149 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:46 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3035296199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:09:47.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.112+0000 7fadd9925700 1 -- 172.21.15.115:0/933746001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4103a10 msgr2=0x7fadd410a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:47.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.112+0000 7fadd9925700 1 --2- 172.21.15.115:0/933746001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4103a10 0x7fadd410a340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00b0a0 tx=0x7fadcc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.112+0000 7fadd9925700 1 -- 172.21.15.115:0/933746001 shutdown_connections 2023-07-14T18:09:47.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.112+0000 7fadd9925700 1 --2- 172.21.15.115:0/933746001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4103a10 0x7fadd410a340 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.112+0000 7fadd9925700 1 --2- 172.21.15.115:0/933746001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd4101e80 0x7fadd4102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.112+0000 7fadd9925700 1 --2- 172.21.15.115:0/933746001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd4103060 0x7fadd41034d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.112+0000 7fadd9925700 1 -- 172.21.15.115:0/933746001 >> 172.21.15.115:0/933746001 conn(0x7fadd40fd4f0 msgr2=0x7fadd40ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:47.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.112+0000 7fadd9925700 1 -- 172.21.15.115:0/933746001 shutdown_connections 2023-07-14T18:09:47.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.112+0000 7fadd9925700 1 -- 172.21.15.115:0/933746001 wait complete. 2023-07-14T18:09:47.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd9925700 1 Processor -- start 2023-07-14T18:09:47.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd9925700 1 -- start start 2023-07-14T18:09:47.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd9925700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4101e80 0x7fadd4197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:47.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd9925700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd4103060 0x7fadd41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:47.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd9925700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd4103a10 0x7fadd419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:47.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd9925700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd419f720 con 0x7fadd4101e80 2023-07-14T18:09:47.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd9925700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd419f860 con 0x7fadd4103060 2023-07-14T18:09:47.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd9925700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd419f9f0 con 0x7fadd4103a10 2023-07-14T18:09:47.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd2ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4101e80 0x7fadd4197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:47.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd2ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4101e80 0x7fadd4197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52242/0 (socket says 172.21.15.115:52242) 2023-07-14T18:09:47.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.113+0000 7fadd2ffd700 1 -- 172.21.15.115:0/2910815973 learned_addr learned my addr 172.21.15.115:0/2910815973 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:47.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd37fe700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd4103a10 0x7fadd419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:47.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd27fc700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd4103060 0x7fadd41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:47.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd2ffd700 1 -- 172.21.15.115:0/2910815973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd4103a10 msgr2=0x7fadd419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:47.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd2ffd700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd4103a10 0x7fadd419f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd2ffd700 1 -- 172.21.15.115:0/2910815973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd4103060 msgr2=0x7fadd41980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:47.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd2ffd700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd4103060 0x7fadd41980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd2ffd700 1 -- 172.21.15.115:0/2910815973 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc009d00 con 0x7fadd4101e80 2023-07-14T18:09:47.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd27fc700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd4103060 0x7fadd41980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:09:47.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd2ffd700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4101e80 0x7fadd4197ba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fadc400b500 tx=0x7fadc400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:47.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd8923700 1 -- 172.21.15.115:0/2910815973 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4011830 con 0x7fadd4101e80 2023-07-14T18:09:47.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd8923700 1 -- 172.21.15.115:0/2910815973 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fadc4011e20 con 0x7fadd4101e80 2023-07-14T18:09:47.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd419fcd0 con 0x7fadd4101e80 2023-07-14T18:09:47.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd8923700 1 -- 172.21.15.115:0/2910815973 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4010dd0 con 0x7fadd4101e80 2023-07-14T18:09:47.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.114+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd41a0200 con 0x7fadd4101e80 2023-07-14T18:09:47.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.115+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd4108db0 con 0x7fadd4101e80 2023-07-14T18:09:47.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.116+0000 7fadd8923700 1 -- 172.21.15.115:0/2910815973 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fadc4011990 con 0x7fadd4101e80 2023-07-14T18:09:47.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.117+0000 7fadd8923700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fadbc0a1220 0x7fadbc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:47.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.117+0000 7fadd27fc700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fadbc0a1220 0x7fadbc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:47.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.117+0000 7fadd8923700 1 -- 172.21.15.115:0/2910815973 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fadc40cc270 con 0x7fadd4101e80 2023-07-14T18:09:47.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.117+0000 7fadd27fc700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fadbc0a1220 0x7fadbc0a36d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fadc8005fd0 tx=0x7fadc8005e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:47.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.121+0000 7fadd8923700 1 -- 172.21.15.115:0/2910815973 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fadc4095e60 con 0x7fadd4101e80 2023-07-14T18:09:47.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:46 smithi115 ceph-mon[127451]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi115=a;smithi115=d;smithi115=g;smithi115=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:09:47.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:46 smithi115 ceph-mon[127451]: Saving service mds.all spec with placement smithi115=a;smithi115=d;smithi115=g;smithi115=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l;count:12 2023-07-14T18:09:47.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:46 smithi115 ceph-mon[127451]: Deploying daemon mds.l on smithi187 2023-07-14T18:09:47.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:46 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3035296199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:09:47.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:46 smithi150 ceph-mon[128853]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi115=a;smithi115=d;smithi115=g;smithi115=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:09:47.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:46 smithi150 ceph-mon[128853]: Saving service mds.all spec with placement smithi115=a;smithi115=d;smithi115=g;smithi115=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l;count:12 2023-07-14T18:09:47.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:46 smithi150 ceph-mon[128853]: Deploying daemon mds.l on smithi187 2023-07-14T18:09:47.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:46 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3035296199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:09:47.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.351+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadd402cd00 con 0x7fadd4101e80 2023-07-14T18:09:47.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.352+0000 7fadd8923700 1 -- 172.21.15.115:0/2910815973 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fadc4015210 con 0x7fadd4101e80 2023-07-14T18:09:47.388 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:47.389 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-14T18:09:47.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.353+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fadbc0a1220 msgr2=0x7fadbc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:47.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.353+0000 7fadd9925700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fadbc0a1220 0x7fadbc0a36d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fadc8005fd0 tx=0x7fadc8005e40 comp rx=0 tx=0).stop 2023-07-14T18:09:47.390 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.353+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4101e80 msgr2=0x7fadd4197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:47.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.353+0000 7fadd9925700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4101e80 0x7fadd4197ba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fadc400b500 tx=0x7fadc400b810 comp rx=0 tx=0).stop 2023-07-14T18:09:47.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.354+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 shutdown_connections 2023-07-14T18:09:47.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.354+0000 7fadd9925700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fadbc0a1220 0x7fadbc0a36d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.354+0000 7fadd9925700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadd4103a10 0x7fadd419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.354+0000 7fadd9925700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fadd4101e80 0x7fadd4197ba0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.354+0000 7fadd9925700 1 --2- 172.21.15.115:0/2910815973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadd4103060 0x7fadd41980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:47.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.354+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 >> 172.21.15.115:0/2910815973 conn(0x7fadd40fd4f0 msgr2=0x7fadd40ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:47.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.354+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 shutdown_connections 2023-07-14T18:09:47.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:47.354+0000 7fadd9925700 1 -- 172.21.15.115:0/2910815973 wait complete. 2023-07-14T18:09:47.392 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 1 2023-07-14T18:09:47.432 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:09:47 smithi187 systemd[1]: Starting Ceph mds.l for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:09:47.809 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:09:47 smithi187 podman[154445]: 2023-07-14T18:09:47.809 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mds[154485]: -- 172.21.15.187:0/838689379 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5560018231e0 con 0x5560017f8c00 2023-07-14T18:09:47.810 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-l[154461]: starting mds.l at 2023-07-14T18:09:48.182 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:09:47 smithi187 bash[154445]: 145fe675c3d58beb63963e780a392d3bcc21ef473c67a994b55f8ed8a19edb56 2023-07-14T18:09:48.183 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:09:47 smithi187 systemd[1]: Started Ceph mds.l for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:09:48.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:48.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3769573160' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:09:48.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2910815973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:48.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.187:6832/1471546028,v1:172.21.15.187:6833/1471546028] up:boot 2023-07-14T18:09:48.184 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: fsmap 1 up:standby 2023-07-14T18:09:48.184 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T18:09:48.184 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:48.184 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:48.184 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:48.184 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:48.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:48.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:47 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:48.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:48.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3769573160' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:09:48.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2910815973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:48.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.187:6832/1471546028,v1:172.21.15.187:6833/1471546028] up:boot 2023-07-14T18:09:48.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: fsmap 1 up:standby 2023-07-14T18:09:48.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T18:09:48.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:48.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:48.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:48.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:48.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:48.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:47 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:48.369 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:48.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3769573160' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:09:48.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2910815973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:48.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.187:6832/1471546028,v1:172.21.15.187:6833/1471546028] up:boot 2023-07-14T18:09:48.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: fsmap 1 up:standby 2023-07-14T18:09:48.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T18:09:48.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:48.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:48.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:48.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:48.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:48.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:47 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:49.118 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:48 smithi150 ceph-mon[128853]: Deploying daemon mds.k on smithi150 2023-07-14T18:09:49.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:48 smithi115 ceph-mon[127451]: Deploying daemon mds.k on smithi150 2023-07-14T18:09:49.403 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:09:49.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:48 smithi187 ceph-mon[129317]: Deploying daemon mds.k on smithi150 2023-07-14T18:09:49.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/389117980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b34100680 msgr2=0x7f2b34106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:49.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/389117980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b34100680 0x7f2b34106f40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00b0a0 tx=0x7f2b2c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/389117980 shutdown_connections 2023-07-14T18:09:49.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/389117980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b34100680 0x7f2b34106f40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/389117980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b340feb00 0x7f2b340fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/389117980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b340ffcf0 0x7f2b34100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/389117980 >> 172.21.15.115:0/389117980 conn(0x7f2b340fa100 msgr2=0x7f2b340fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:49.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/389117980 shutdown_connections 2023-07-14T18:09:49.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/389117980 wait complete. 2023-07-14T18:09:49.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 Processor -- start 2023-07-14T18:09:49.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 -- start start 2023-07-14T18:09:49.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b340feb00 0x7f2b34195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:49.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.504+0000 7f2b3c5fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b340ffcf0 0x7f2b34195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:49.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3c5fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b34100680 0x7f2b3419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:49.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3c5fc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3419d510 con 0x7f2b340feb00 2023-07-14T18:09:49.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3c5fc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3419d650 con 0x7f2b34100680 2023-07-14T18:09:49.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3c5fc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3419d7e0 con 0x7f2b340ffcf0 2023-07-14T18:09:49.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3a398700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b340feb00 0x7f2b34195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:49.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3a398700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b340feb00 0x7f2b34195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52254/0 (socket says 172.21.15.115:52254) 2023-07-14T18:09:49.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3a398700 1 -- 172.21.15.115:0/1220130485 learned_addr learned my addr 172.21.15.115:0/1220130485 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:49.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3ab99700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b34100680 0x7f2b3419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:49.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b39b97700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b340ffcf0 0x7f2b34195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:49.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3a398700 1 -- 172.21.15.115:0/1220130485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b340ffcf0 msgr2=0x7f2b34195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:49.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3a398700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b340ffcf0 0x7f2b34195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3a398700 1 -- 172.21.15.115:0/1220130485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b34100680 msgr2=0x7f2b3419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:49.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3a398700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b34100680 0x7f2b3419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3a398700 1 -- 172.21.15.115:0/1220130485 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c009d00 con 0x7f2b340feb00 2023-07-14T18:09:49.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3a398700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b340feb00 0x7f2b34195990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000e970 tx=0x7f2b3000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:49.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b2b7fe700 1 -- 172.21.15.115:0/1220130485 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30009790 con 0x7f2b340feb00 2023-07-14T18:09:49.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3419dac0 con 0x7f2b340feb00 2023-07-14T18:09:49.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b2b7fe700 1 -- 172.21.15.115:0/1220130485 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2b3000eec0 con 0x7f2b340feb00 2023-07-14T18:09:49.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b2b7fe700 1 -- 172.21.15.115:0/1220130485 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30003c10 con 0x7f2b340feb00 2023-07-14T18:09:49.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.505+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3419dfd0 con 0x7f2b340feb00 2023-07-14T18:09:49.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.506+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b34106040 con 0x7f2b340feb00 2023-07-14T18:09:49.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.508+0000 7f2b2b7fe700 1 -- 172.21.15.115:0/1220130485 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f2b30003d90 con 0x7f2b340feb00 2023-07-14T18:09:49.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.508+0000 7f2b2b7fe700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2b200a11a0 0x7f2b200a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:49.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.508+0000 7f2b2b7fe700 1 -- 172.21.15.115:0/1220130485 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2b30014070 con 0x7f2b340feb00 2023-07-14T18:09:49.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.508+0000 7f2b39b97700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2b200a11a0 0x7f2b200a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:49.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.509+0000 7f2b39b97700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2b200a11a0 0x7f2b200a3650 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2b24006fd0 tx=0x7f2b24008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:49.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.512+0000 7f2b2b7fe700 1 -- 172.21.15.115:0/1220130485 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2b30096440 con 0x7f2b340feb00 2023-07-14T18:09:49.656 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:09:49 smithi150 systemd[1]: Starting Ceph mds.k for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:09:49.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.742+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b3402c8e0 con 0x7f2b340feb00 2023-07-14T18:09:49.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.743+0000 7f2b2b7fe700 1 -- 172.21.15.115:0/1220130485 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f2b30099a60 con 0x7f2b340feb00 2023-07-14T18:09:49.780 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:49.780 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-14T18:09:49.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2b200a11a0 msgr2=0x7f2b200a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:49.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2b200a11a0 0x7f2b200a3650 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2b24006fd0 tx=0x7f2b24008040 comp rx=0 tx=0).stop 2023-07-14T18:09:49.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b340feb00 msgr2=0x7f2b34195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:49.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b340feb00 0x7f2b34195990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000e970 tx=0x7f2b3000ec80 comp rx=0 tx=0).stop 2023-07-14T18:09:49.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 shutdown_connections 2023-07-14T18:09:49.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2b200a11a0 0x7f2b200a3650 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b34100680 0x7f2b3419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b340feb00 0x7f2b34195990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 --2- 172.21.15.115:0/1220130485 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2b340ffcf0 0x7f2b34195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:49.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 >> 172.21.15.115:0/1220130485 conn(0x7f2b340fa100 msgr2=0x7f2b34103180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:49.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 shutdown_connections 2023-07-14T18:09:49.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:49.745+0000 7f2b3c5fc700 1 -- 172.21.15.115:0/1220130485 wait complete. 2023-07-14T18:09:49.783 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 2 2023-07-14T18:09:49.961 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:09:49 smithi150 podman[152975]: 2023-07-14T18:09:49.961 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:09:49 smithi150 bash[152975]: be054a9482459378d6ac25fdb7b47577a3ef103a456341f9c5a719a5a84b58a0 2023-07-14T18:09:49.962 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:09:49 smithi150 ceph-mds[153015]: -- 172.21.15.150:0/988702126 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5629c349b1e0 con 0x5629c3471000 2023-07-14T18:09:49.962 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:09:49 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-k[152991]: starting mds.k at 2023-07-14T18:09:50.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:49 smithi115 ceph-mon[127451]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:50.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:49 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1220130485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:50.369 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:09:49 smithi150 systemd[1]: Started Ceph mds.k for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:09:50.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:49 smithi150 ceph-mon[128853]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:50.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:49 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1220130485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:50.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:49 smithi187 ceph-mon[129317]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:50.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:49 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1220130485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:51.256 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.150:6832/1440309649,v1:172.21.15.150:6833/1440309649] up:boot 2023-07-14T18:09:51.256 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: fsmap 2 up:standby 2023-07-14T18:09:51.256 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T18:09:51.256 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:51.256 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:51.257 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:51.257 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:51.257 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:51.257 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:51.257 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:50 smithi115 ceph-mon[127451]: Deploying daemon mds.j on smithi115 2023-07-14T18:09:51.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.150:6832/1440309649,v1:172.21.15.150:6833/1440309649] up:boot 2023-07-14T18:09:51.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: fsmap 2 up:standby 2023-07-14T18:09:51.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T18:09:51.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:51.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:51.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:51.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:51.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:51.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:51.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:50 smithi150 ceph-mon[128853]: Deploying daemon mds.j on smithi115 2023-07-14T18:09:51.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.150:6832/1440309649,v1:172.21.15.150:6833/1440309649] up:boot 2023-07-14T18:09:51.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: fsmap 2 up:standby 2023-07-14T18:09:51.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T18:09:51.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:51.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:51.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:51.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:51.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:51.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:51.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:50 smithi187 ceph-mon[129317]: Deploying daemon mds.j on smithi115 2023-07-14T18:09:51.605 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:09:51 smithi115 systemd[1]: Starting Ceph mds.j for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:09:51.794 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:09:51.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.888+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/968632267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df4100680 msgr2=0x7f9df4106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:51.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.888+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/968632267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df4100680 0x7f9df4106f40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9de400b0a0 tx=0x7f9de400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:09:51.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.888+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/968632267 shutdown_connections 2023-07-14T18:09:51.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.888+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/968632267 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df4100680 0x7f9df4106f40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:51.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.888+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/968632267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df40feb00 0x7f9df40fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:51.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.888+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/968632267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df40ffcf0 0x7f9df4100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:51.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.888+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/968632267 >> 172.21.15.115:0/968632267 conn(0x7f9df40fa100 msgr2=0x7f9df40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:51.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.888+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/968632267 shutdown_connections 2023-07-14T18:09:51.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.888+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/968632267 wait complete. 2023-07-14T18:09:51.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.889+0000 7f9dfbb5d700 1 Processor -- start 2023-07-14T18:09:51.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.889+0000 7f9dfbb5d700 1 -- start start 2023-07-14T18:09:51.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.889+0000 7f9dfbb5d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df40feb00 0x7f9df4194880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:51.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.889+0000 7f9dfbb5d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df40ffcf0 0x7f9df4194dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:51.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.889+0000 7f9dfbb5d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df4100680 0x7f9df419be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:51.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9dfbb5d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df419c400 con 0x7f9df40feb00 2023-07-14T18:09:51.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9dfbb5d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df419c540 con 0x7f9df4100680 2023-07-14T18:09:51.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9dfbb5d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df419c6d0 con 0x7f9df40ffcf0 2023-07-14T18:09:51.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df98f9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df40feb00 0x7f9df4194880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:51.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df98f9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df40feb00 0x7f9df4194880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52268/0 (socket says 172.21.15.115:52268) 2023-07-14T18:09:51.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df98f9700 1 -- 172.21.15.115:0/3043014052 learned_addr learned my addr 172.21.15.115:0/3043014052 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:51.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df90f8700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df40ffcf0 0x7f9df4194dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:51.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9dfa0fa700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df4100680 0x7f9df419be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:51.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df98f9700 1 -- 172.21.15.115:0/3043014052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df40ffcf0 msgr2=0x7f9df4194dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:51.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df98f9700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df40ffcf0 0x7f9df4194dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:51.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df98f9700 1 -- 172.21.15.115:0/3043014052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df4100680 msgr2=0x7f9df419be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:51.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df98f9700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df4100680 0x7f9df419be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:51.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df98f9700 1 -- 172.21.15.115:0/3043014052 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de4009d00 con 0x7f9df40feb00 2023-07-14T18:09:51.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df98f9700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df40feb00 0x7f9df4194880 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00d980 tx=0x7f9dec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:51.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df2ffd700 1 -- 172.21.15.115:0/3043014052 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00b950 con 0x7f9df40feb00 2023-07-14T18:09:51.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df2ffd700 1 -- 172.21.15.115:0/3043014052 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9dec007620 con 0x7f9df40feb00 2023-07-14T18:09:51.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9df2ffd700 1 -- 172.21.15.115:0/3043014052 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00f3c0 con 0x7f9df40feb00 2023-07-14T18:09:51.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df419c9b0 con 0x7f9df40feb00 2023-07-14T18:09:51.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.890+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df419ce70 con 0x7f9df40feb00 2023-07-14T18:09:51.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.891+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df410a1e0 con 0x7f9df40feb00 2023-07-14T18:09:51.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.892+0000 7f9df2ffd700 1 -- 172.21.15.115:0/3043014052 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f9dec00bab0 con 0x7f9df40feb00 2023-07-14T18:09:51.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.893+0000 7f9df2ffd700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11a0 0x7f9ddc0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:51.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.893+0000 7f9df90f8700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11a0 0x7f9ddc0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:51.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.893+0000 7f9df2ffd700 1 -- 172.21.15.115:0/3043014052 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9dec0ccba0 con 0x7f9df40feb00 2023-07-14T18:09:51.936 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.893+0000 7f9df90f8700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11a0 0x7f9ddc0a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005950 tx=0x7f9de800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:51.937 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:51.896+0000 7f9df2ffd700 1 -- 172.21.15.115:0/3043014052 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9dec0cce60 con 0x7f9df40feb00 2023-07-14T18:09:52.028 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:09:51 smithi115 podman[160777]: 2023-07-14T18:09:52.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.157+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9df410a3d0 con 0x7f9df40feb00 2023-07-14T18:09:52.194 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.157+0000 7f9df2ffd700 1 -- 172.21.15.115:0/3043014052 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2780 (secure 0 0 0) 0x7f9dec096790 con 0x7f9df40feb00 2023-07-14T18:09:52.194 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:52.195 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-14T18:09:52.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11a0 msgr2=0x7f9ddc0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:52.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11a0 0x7f9ddc0a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005950 tx=0x7f9de800b410 comp rx=0 tx=0).stop 2023-07-14T18:09:52.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df40feb00 msgr2=0x7f9df4194880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:52.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df40feb00 0x7f9df4194880 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00d980 tx=0x7f9dec00dc90 comp rx=0 tx=0).stop 2023-07-14T18:09:52.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 shutdown_connections 2023-07-14T18:09:52.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11a0 0x7f9ddc0a3650 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:52.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df4100680 0x7f9df419be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:52.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df40feb00 0x7f9df4194880 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:52.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 --2- 172.21.15.115:0/3043014052 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df40ffcf0 0x7f9df4194dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:52.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 >> 172.21.15.115:0/3043014052 conn(0x7f9df40fa100 msgr2=0x7f9df40fc320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:52.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 shutdown_connections 2023-07-14T18:09:52.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:52.160+0000 7f9dfbb5d700 1 -- 172.21.15.115:0/3043014052 wait complete. 2023-07-14T18:09:52.198 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 4 2023-07-14T18:09:52.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:52 smithi115 ceph-mon[127451]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:52.355 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:09:51 smithi115 ceph-mds[160859]: -- 172.21.15.115:0/4014596116 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fcabf4d1e0 con 0x55fcabf23000 2023-07-14T18:09:52.355 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:09:51 smithi115 bash[160777]: ab53730f3665cd31c038a59d38a1500421d5ad5d3a6f4b72360aa907bb6a9207 2023-07-14T18:09:52.355 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:09:51 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-j[160815]: starting mds.j at 2023-07-14T18:09:52.355 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:09:52 smithi115 systemd[1]: Started Ceph mds.j for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:09:52.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:52 smithi150 ceph-mon[128853]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:52.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:52 smithi187 ceph-mon[129317]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:53.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.115:6834/3105890615,v1:172.21.15.115:6835/3105890615] up:boot 2023-07-14T18:09:53.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: fsmap 3 up:standby 2023-07-14T18:09:53.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T18:09:53.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:53.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:53.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:53.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:53.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:53.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:53.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: Deploying daemon mds.d on smithi115 2023-07-14T18:09:53.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:53 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3043014052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:53.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.115:6834/3105890615,v1:172.21.15.115:6835/3105890615] up:boot 2023-07-14T18:09:53.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: fsmap 3 up:standby 2023-07-14T18:09:53.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T18:09:53.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:53.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:53.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:53.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:53.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:53.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:53.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: Deploying daemon mds.d on smithi115 2023-07-14T18:09:53.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:53 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3043014052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:53.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.115:6834/3105890615,v1:172.21.15.115:6835/3105890615] up:boot 2023-07-14T18:09:53.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: fsmap 3 up:standby 2023-07-14T18:09:53.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T18:09:53.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:53.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:53.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:53.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:53.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:53.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:53.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: Deploying daemon mds.d on smithi115 2023-07-14T18:09:53.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:53 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3043014052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:53.993 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:09:53 smithi115 systemd[1]: Starting Ceph mds.d for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:09:54.211 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:09:54.245 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:54 smithi115 ceph-mon[127451]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:54.245 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:09:53 smithi115 podman[161160]: 2023-07-14T18:09:54.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.307+0000 7f4154e23700 1 -- 172.21.15.115:0/3926834820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150101790 msgr2=0x7f4150108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:54.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.307+0000 7f4154e23700 1 --2- 172.21.15.115:0/3926834820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150101790 0x7f4150108050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f414800b0a0 tx=0x7f414800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.307+0000 7f4154e23700 1 -- 172.21.15.115:0/3926834820 shutdown_connections 2023-07-14T18:09:54.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.307+0000 7f4154e23700 1 --2- 172.21.15.115:0/3926834820 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150101790 0x7f4150108050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.307+0000 7f4154e23700 1 --2- 172.21.15.115:0/3926834820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41500ffc10 0x7f4150100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.307+0000 7f4154e23700 1 --2- 172.21.15.115:0/3926834820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4150100e00 0x7f4150101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.307+0000 7f4154e23700 1 -- 172.21.15.115:0/3926834820 >> 172.21.15.115:0/3926834820 conn(0x7f41500fb230 msgr2=0x7f41500fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:54.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.307+0000 7f4154e23700 1 -- 172.21.15.115:0/3926834820 shutdown_connections 2023-07-14T18:09:54.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.307+0000 7f4154e23700 1 -- 172.21.15.115:0/3926834820 wait complete. 2023-07-14T18:09:54.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f4154e23700 1 Processor -- start 2023-07-14T18:09:54.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f4154e23700 1 -- start start 2023-07-14T18:09:54.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f4154e23700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41500ffc10 0x7f41500743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:54.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f4154e23700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150100e00 0x7f4150072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:54.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f4154e23700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4150101790 0x7f4150072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:54.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f4154e23700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4150073710 con 0x7f4150100e00 2023-07-14T18:09:54.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f4154e23700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4150073880 con 0x7f4150101790 2023-07-14T18:09:54.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f4154e23700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415019f960 con 0x7f41500ffc10 2023-07-14T18:09:54.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f414dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150100e00 0x7f4150072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:54.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f414dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150100e00 0x7f4150072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52306/0 (socket says 172.21.15.115:52306) 2023-07-14T18:09:54.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f414dd9b700 1 -- 172.21.15.115:0/704230188 learned_addr learned my addr 172.21.15.115:0/704230188 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:54.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f414e59c700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41500ffc10 0x7f41500743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:54.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f414ed9d700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4150101790 0x7f4150072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:54.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f414dd9b700 1 -- 172.21.15.115:0/704230188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41500ffc10 msgr2=0x7f41500743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:54.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.308+0000 7f414dd9b700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41500ffc10 0x7f41500743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f414dd9b700 1 -- 172.21.15.115:0/704230188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4150101790 msgr2=0x7f4150072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:54.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f414dd9b700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4150101790 0x7f4150072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f414dd9b700 1 -- 172.21.15.115:0/704230188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4148009d00 con 0x7f4150100e00 2023-07-14T18:09:54.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f414e59c700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41500ffc10 0x7f41500743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:09:54.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f414ed9d700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4150101790 0x7f4150072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:09:54.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f414dd9b700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150100e00 0x7f4150072a30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f414400d950 tx=0x7f414400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:54.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f413f7fe700 1 -- 172.21.15.115:0/704230188 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414400f890 con 0x7f4150100e00 2023-07-14T18:09:54.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f413f7fe700 1 -- 172.21.15.115:0/704230188 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f414400fe80 con 0x7f4150100e00 2023-07-14T18:09:54.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415019fc40 con 0x7f4150100e00 2023-07-14T18:09:54.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f413f7fe700 1 -- 172.21.15.115:0/704230188 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414400e3b0 con 0x7f4150100e00 2023-07-14T18:09:54.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.309+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41501a0150 con 0x7f4150100e00 2023-07-14T18:09:54.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.311+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4150107170 con 0x7f4150100e00 2023-07-14T18:09:54.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.311+0000 7f413f7fe700 1 -- 172.21.15.115:0/704230188 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f4144010460 con 0x7f4150100e00 2023-07-14T18:09:54.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.311+0000 7f413f7fe700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41380a11a0 0x7f41380a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:54.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.312+0000 7f414e59c700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41380a11a0 0x7f41380a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:54.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.312+0000 7f413f7fe700 1 -- 172.21.15.115:0/704230188 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f41440cc210 con 0x7f4150100e00 2023-07-14T18:09:54.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.312+0000 7f414e59c700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41380a11a0 0x7f41380a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4150100c60 tx=0x7f414000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:54.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.316+0000 7f413f7fe700 1 -- 172.21.15.115:0/704230188 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4144095e00 con 0x7f4150100e00 2023-07-14T18:09:54.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:54 smithi150 ceph-mon[128853]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:54.432 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:54 smithi187 ceph-mon[129317]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:54.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.550+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f415002c8e0 con 0x7f4150100e00 2023-07-14T18:09:54.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.550+0000 7f413f7fe700 1 -- 172.21.15.115:0/704230188 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2780 (secure 0 0 0) 0x7f4144099420 con 0x7f4150100e00 2023-07-14T18:09:54.587 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:54.588 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-14T18:09:54.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41380a11a0 msgr2=0x7f41380a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:54.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41380a11a0 0x7f41380a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4150100c60 tx=0x7f414000b410 comp rx=0 tx=0).stop 2023-07-14T18:09:54.589 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150100e00 msgr2=0x7f4150072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:54.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150100e00 0x7f4150072a30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f414400d950 tx=0x7f414400dc60 comp rx=0 tx=0).stop 2023-07-14T18:09:54.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 shutdown_connections 2023-07-14T18:09:54.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41380a11a0 0x7f41380a3650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4150101790 0x7f4150072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41500ffc10 0x7f41500743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 --2- 172.21.15.115:0/704230188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4150100e00 0x7f4150072a30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:54.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.552+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 >> 172.21.15.115:0/704230188 conn(0x7f41500fb230 msgr2=0x7f41501042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:54.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.553+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 shutdown_connections 2023-07-14T18:09:54.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:54.553+0000 7f4154e23700 1 -- 172.21.15.115:0/704230188 wait complete. 2023-07-14T18:09:54.591 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 4 2023-07-14T18:09:54.600 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:09:54 smithi115 bash[161160]: 093c77f671f5e73621ecc3449ccba48af7339059bb408642cc001a01ea3f86c1 2023-07-14T18:09:54.600 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:09:54 smithi115 ceph-mds[161211]: -- 172.21.15.115:0/1285244172 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556d5ba811e0 con 0x556d5ba56800 2023-07-14T18:09:54.601 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:09:54 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-d[161176]: starting mds.d at 2023-07-14T18:09:54.601 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:09:54 smithi115 systemd[1]: Started Ceph mds.d for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:09:55.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:55.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:55.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:55.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:55.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:55.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:55.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: Deploying daemon mds.e on smithi150 2023-07-14T18:09:55.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/704230188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:55.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:55.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.115:6836/750049188,v1:172.21.15.115:6837/750049188] up:boot 2023-07-14T18:09:55.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: fsmap 4 up:standby 2023-07-14T18:09:55.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:55 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T18:09:55.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:55.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:55.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:55.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:55.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:55.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:55.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: Deploying daemon mds.e on smithi150 2023-07-14T18:09:55.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/704230188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:55.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:55.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.115:6836/750049188,v1:172.21.15.115:6837/750049188] up:boot 2023-07-14T18:09:55.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: fsmap 4 up:standby 2023-07-14T18:09:55.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:55 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T18:09:55.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:55.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:55.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:55.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:55.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:55.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:55.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: Deploying daemon mds.e on smithi150 2023-07-14T18:09:55.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/704230188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:55.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:55.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.115:6836/750049188,v1:172.21.15.115:6837/750049188] up:boot 2023-07-14T18:09:55.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: fsmap 4 up:standby 2023-07-14T18:09:55.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:55 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T18:09:56.029 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:09:55 smithi150 systemd[1]: Starting Ceph mds.e for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:09:56.337 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:09:55 smithi150 podman[153325]: 2023-07-14T18:09:56.337 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:09:56 smithi150 ceph-mds[153365]: -- 172.21.15.150:0/337399199 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5556b58ef1e0 con 0x5556b58c4c00 2023-07-14T18:09:56.338 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:09:56 smithi150 bash[153325]: 65c51609c3d97a435309c60c900a85d885f54246f56e7aed988f63438b8618fb 2023-07-14T18:09:56.338 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:09:56 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-e[153341]: starting mds.e at 2023-07-14T18:09:56.602 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:09:56.619 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:09:56 smithi150 systemd[1]: Started Ceph mds.e for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:09:56.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.701+0000 7f35ee640700 1 -- 172.21.15.115:0/3625295476 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80ffd30 msgr2=0x7f35e8100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:56.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.701+0000 7f35ee640700 1 --2- 172.21.15.115:0/3625295476 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80ffd30 0x7f35e8100180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009990 tx=0x7f35dc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.701+0000 7f35ee640700 1 -- 172.21.15.115:0/3625295476 shutdown_connections 2023-07-14T18:09:56.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.701+0000 7f35ee640700 1 --2- 172.21.15.115:0/3625295476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35e81006c0 0x7f35e8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.701+0000 7f35ee640700 1 --2- 172.21.15.115:0/3625295476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35e80feb40 0x7f35e80fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.701+0000 7f35ee640700 1 --2- 172.21.15.115:0/3625295476 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80ffd30 0x7f35e8100180 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.741 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.701+0000 7f35ee640700 1 -- 172.21.15.115:0/3625295476 >> 172.21.15.115:0/3625295476 conn(0x7f35e80fa160 msgr2=0x7f35e80fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:56.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 -- 172.21.15.115:0/3625295476 shutdown_connections 2023-07-14T18:09:56.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 -- 172.21.15.115:0/3625295476 wait complete. 2023-07-14T18:09:56.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 Processor -- start 2023-07-14T18:09:56.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 -- start start 2023-07-14T18:09:56.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80feb40 0x7f35e80692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:56.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35e80ffd30 0x7f35e819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:56.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35e81006c0 0x7f35e819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:56.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e819d3b0 con 0x7f35e80feb40 2023-07-14T18:09:56.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e819d520 con 0x7f35e80ffd30 2023-07-14T18:09:56.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.702+0000 7f35ee640700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e819d6e0 con 0x7f35e81006c0 2023-07-14T18:09:56.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80feb40 0x7f35e80692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:56.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80feb40 0x7f35e80692e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52320/0 (socket says 172.21.15.115:52320) 2023-07-14T18:09:56.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e7fff700 1 -- 172.21.15.115:0/1765197980 learned_addr learned my addr 172.21.15.115:0/1765197980 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:56.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35ecbdd700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35e81006c0 0x7f35e819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:56.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e77fe700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35e80ffd30 0x7f35e819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:56.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e7fff700 1 -- 172.21.15.115:0/1765197980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35e81006c0 msgr2=0x7f35e819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:56.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e7fff700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35e81006c0 0x7f35e819cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e7fff700 1 -- 172.21.15.115:0/1765197980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35e80ffd30 msgr2=0x7f35e819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:56.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e7fff700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35e80ffd30 0x7f35e819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e7fff700 1 -- 172.21.15.115:0/1765197980 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc0095f0 con 0x7f35e80feb40 2023-07-14T18:09:56.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e7fff700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80feb40 0x7f35e80692e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f35d800b500 tx=0x7f35d800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:56.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e57fa700 1 -- 172.21.15.115:0/1765197980 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8011830 con 0x7f35e80feb40 2023-07-14T18:09:56.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e57fa700 1 -- 172.21.15.115:0/1765197980 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f35d8011e20 con 0x7f35e80feb40 2023-07-14T18:09:56.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e819d9c0 con 0x7f35e80feb40 2023-07-14T18:09:56.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35e57fa700 1 -- 172.21.15.115:0/1765197980 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8010dd0 con 0x7f35e80feb40 2023-07-14T18:09:56.746 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.703+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e819ded0 con 0x7f35e80feb40 2023-07-14T18:09:56.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.704+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e810a260 con 0x7f35e80feb40 2023-07-14T18:09:56.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.705+0000 7f35e57fa700 1 -- 172.21.15.115:0/1765197980 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f35d8011990 con 0x7f35e80feb40 2023-07-14T18:09:56.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.706+0000 7f35e57fa700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35d00a11a0 0x7f35d00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:56.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.706+0000 7f35e77fe700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35d00a11a0 0x7f35d00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:56.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.706+0000 7f35e57fa700 1 -- 172.21.15.115:0/1765197980 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f35d80cc2d0 con 0x7f35e80feb40 2023-07-14T18:09:56.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.706+0000 7f35e77fe700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35d00a11a0 0x7f35d00a3650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f35dc00c010 tx=0x7f35dc00b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:56.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.709+0000 7f35e57fa700 1 -- 172.21.15.115:0/1765197980 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f35d8095ec0 con 0x7f35e80feb40 2023-07-14T18:09:56.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.939+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35e810a420 con 0x7f35e80feb40 2023-07-14T18:09:56.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.939+0000 7f35e57fa700 1 -- 172.21.15.115:0/1765197980 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4297 (secure 0 0 0) 0x7f35d80994e0 con 0x7f35e80feb40 2023-07-14T18:09:56.976 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:56.976 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T18:09:56.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35d00a11a0 msgr2=0x7f35d00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:56.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35d00a11a0 0x7f35d00a3650 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f35dc00c010 tx=0x7f35dc00b540 comp rx=0 tx=0).stop 2023-07-14T18:09:56.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80feb40 msgr2=0x7f35e80692e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:56.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80feb40 0x7f35e80692e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f35d800b500 tx=0x7f35d800b810 comp rx=0 tx=0).stop 2023-07-14T18:09:56.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 shutdown_connections 2023-07-14T18:09:56.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35d00a11a0 0x7f35d00a3650 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35e81006c0 0x7f35e819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35e80feb40 0x7f35e80692e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 --2- 172.21.15.115:0/1765197980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35e80ffd30 0x7f35e819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:56.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 >> 172.21.15.115:0/1765197980 conn(0x7f35e80fa160 msgr2=0x7f35e80fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:56.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 shutdown_connections 2023-07-14T18:09:56.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:56.941+0000 7f35ee640700 1 -- 172.21.15.115:0/1765197980 wait complete. 2023-07-14T18:09:56.980 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 6 2023-07-14T18:09:57.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:57.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:57.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.150:6834/3630781789,v1:172.21.15.150:6835/3630781789] up:boot 2023-07-14T18:09:57.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: fsmap 5 up:standby 2023-07-14T18:09:57.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:09:57.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:57.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:57.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:57.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:57.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: Deploying daemon mds.f on smithi187 2023-07-14T18:09:57.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:57.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:57 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1765197980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:57.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:57.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:57.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.150:6834/3630781789,v1:172.21.15.150:6835/3630781789] up:boot 2023-07-14T18:09:57.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: fsmap 5 up:standby 2023-07-14T18:09:57.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:09:57.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:57.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:57.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:57.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:57.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: Deploying daemon mds.f on smithi187 2023-07-14T18:09:57.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:57.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:57 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1765197980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:57.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:57.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:57.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.150:6834/3630781789,v1:172.21.15.150:6835/3630781789] up:boot 2023-07-14T18:09:57.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: fsmap 5 up:standby 2023-07-14T18:09:57.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:09:57.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:57.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:57.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:57.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:57.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: Deploying daemon mds.f on smithi187 2023-07-14T18:09:57.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:57.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:57 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1765197980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:57.933 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:09:57 smithi187 systemd[1]: Starting Ceph mds.f for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:09:58.432 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:09:58 smithi187 podman[154783]: 2023-07-14T18:09:58.433 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:09:58 smithi187 ceph-mds[154823]: -- 172.21.15.187:0/3293848760 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55964e7871e0 con 0x55964e75c800 2023-07-14T18:09:58.433 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:09:58 smithi187 bash[154783]: 8fe40feddcf997daecda91a6dc522c1a0f7c4d14e44b37412b7dfd0ca4bf63f3 2023-07-14T18:09:58.433 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:09:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-f[154799]: starting mds.f at 2023-07-14T18:09:58.932 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:09:58 smithi187 systemd[1]: Started Ceph mds.f for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:09:58.991 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:09:59.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.088+0000 7f7960390700 1 -- 172.21.15.115:0/1658341005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580fed60 msgr2=0x7f79580ff1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.088+0000 7f7960390700 1 --2- 172.21.15.115:0/1658341005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580fed60 0x7f79580ff1d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7948009990 tx=0x7f7948009ca0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.088+0000 7f7960390700 1 -- 172.21.15.115:0/1658341005 shutdown_connections 2023-07-14T18:09:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 --2- 172.21.15.115:0/1658341005 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580ff710 0x7f7958106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 --2- 172.21.15.115:0/1658341005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7958100110 0x7f7958100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 --2- 172.21.15.115:0/1658341005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580fed60 0x7f79580ff1d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 -- 172.21.15.115:0/1658341005 >> 172.21.15.115:0/1658341005 conn(0x7f79580f98a0 msgr2=0x7f79580fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 -- 172.21.15.115:0/1658341005 shutdown_connections 2023-07-14T18:09:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 -- 172.21.15.115:0/1658341005 wait complete. 2023-07-14T18:09:59.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 Processor -- start 2023-07-14T18:09:59.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 -- start start 2023-07-14T18:09:59.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580fed60 0x7f7958102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:59.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580ff710 0x7f7958101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:59.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7958100110 0x7f7958101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:59.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7958101e30 con 0x7f79580ff710 2023-07-14T18:09:59.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7958101fa0 con 0x7f7958100110 2023-07-14T18:09:59.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f7960390700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795819f900 con 0x7f79580fed60 2023-07-14T18:09:59.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f795d92b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580ff710 0x7f7958101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:59.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.089+0000 7f795d92b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580ff710 0x7f7958101150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52336/0 (socket says 172.21.15.115:52336) 2023-07-14T18:09:59.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f795d92b700 1 -- 172.21.15.115:0/2840838672 learned_addr learned my addr 172.21.15.115:0/2840838672 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:09:59.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f795e12c700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580fed60 0x7f7958102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:59.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f795e92d700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7958100110 0x7f7958101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:59.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f795d92b700 1 -- 172.21.15.115:0/2840838672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580fed60 msgr2=0x7f7958102b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:59.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f795d92b700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580fed60 0x7f7958102b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.130 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f795d92b700 1 -- 172.21.15.115:0/2840838672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7958100110 msgr2=0x7f7958101690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:59.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f795d92b700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7958100110 0x7f7958101690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f795d92b700 1 -- 172.21.15.115:0/2840838672 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79480095f0 con 0x7f79580ff710 2023-07-14T18:09:59.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f795d92b700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580ff710 0x7f7958101150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7948005950 tx=0x7f7948009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:59.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f794f7fe700 1 -- 172.21.15.115:0/2840838672 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794801d070 con 0x7f79580ff710 2023-07-14T18:09:59.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795819fb80 con 0x7f79580ff710 2023-07-14T18:09:59.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f794f7fe700 1 -- 172.21.15.115:0/2840838672 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f794800bd70 con 0x7f79580ff710 2023-07-14T18:09:59.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f794f7fe700 1 -- 172.21.15.115:0/2840838672 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948022d40 con 0x7f79580ff710 2023-07-14T18:09:59.131 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.090+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795819ffa0 con 0x7f79580ff710 2023-07-14T18:09:59.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.091+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795810a1e0 con 0x7f79580ff710 2023-07-14T18:09:59.132 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.092+0000 7f794f7fe700 1 -- 172.21.15.115:0/2840838672 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7948022470 con 0x7f79580ff710 2023-07-14T18:09:59.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.093+0000 7f794f7fe700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79440a11a0 0x7f79440a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:09:59.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.093+0000 7f795e12c700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79440a11a0 0x7f79440a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:09:59.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.093+0000 7f794f7fe700 1 -- 172.21.15.115:0/2840838672 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f79480cd970 con 0x7f79580ff710 2023-07-14T18:09:59.133 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.093+0000 7f795e12c700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79440a11a0 0x7f79440a3650 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7958067580 tx=0x7f795400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:09:59.134 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.097+0000 7f794f7fe700 1 -- 172.21.15.115:0/2840838672 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f79480975f0 con 0x7f79580ff710 2023-07-14T18:09:59.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.333+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79581a05b0 con 0x7f79580ff710 2023-07-14T18:09:59.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.333+0000 7f794f7fe700 1 -- 172.21.15.115:0/2840838672 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5057 (secure 0 0 0) 0x7f7948005c00 con 0x7f79580ff710 2023-07-14T18:09:59.370 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:09:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-07-14T18:09:59.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79440a11a0 msgr2=0x7f79440a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:59.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79440a11a0 0x7f79440a3650 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7958067580 tx=0x7f795400b410 comp rx=0 tx=0).stop 2023-07-14T18:09:59.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580ff710 msgr2=0x7f7958101150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:09:59.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580ff710 0x7f7958101150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7948005950 tx=0x7f7948009580 comp rx=0 tx=0).stop 2023-07-14T18:09:59.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 shutdown_connections 2023-07-14T18:09:59.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79440a11a0 0x7f79440a3650 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7958100110 0x7f7958101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79580fed60 0x7f7958102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 --2- 172.21.15.115:0/2840838672 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79580ff710 0x7f7958101150 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:09:59.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 >> 172.21.15.115:0/2840838672 conn(0x7f79580f98a0 msgr2=0x7f79580fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:09:59.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 shutdown_connections 2023-07-14T18:09:59.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:09:59.335+0000 7f7960390700 1 -- 172.21.15.115:0/2840838672 wait complete. 2023-07-14T18:09:59.374 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 7 2023-07-14T18:09:59.783 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] up:boot 2023-07-14T18:09:59.784 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: fsmap 6 up:standby 2023-07-14T18:09:59.784 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:09:59.784 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:59.784 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:59.784 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:59.784 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:59.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:59.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:59.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: Deploying daemon mds.c on smithi187 2023-07-14T18:09:59.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:59.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:09:59 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2840838672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:59.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] up:boot 2023-07-14T18:09:59.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: fsmap 6 up:standby 2023-07-14T18:09:59.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:09:59.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:59.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:59.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:59.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:59.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:59.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:59.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: Deploying daemon mds.c on smithi187 2023-07-14T18:09:59.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:59.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:09:59 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2840838672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:09:59.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] up:boot 2023-07-14T18:09:59.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: fsmap 6 up:standby 2023-07-14T18:09:59.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:09:59.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:59.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:59.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:09:59.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:09:59.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:09:59.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:09:59.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: Deploying daemon mds.c on smithi187 2023-07-14T18:09:59.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:09:59.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:09:59 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2840838672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:00.054 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:09:59 smithi187 systemd[1]: Starting Ceph mds.c for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:10:00.574 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:00 smithi187 ceph-mon[129317]: overall HEALTH_OK 2023-07-14T18:10:00.574 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:10:00 smithi187 podman[155125]: 2023-07-14T18:10:00.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:00 smithi115 ceph-mon[127451]: overall HEALTH_OK 2023-07-14T18:10:00.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:00 smithi150 ceph-mon[128853]: overall HEALTH_OK 2023-07-14T18:10:00.932 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:10:00 smithi187 bash[155125]: c46fafde96e74f22834bc70c2a6c1ad8c351fe23314bc7741c655feabde559ed 2023-07-14T18:10:00.933 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:10:00 smithi187 ceph-mds[155165]: -- 172.21.15.187:0/719200741 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5604fb21d1e0 con 0x5604fb1f2800 2023-07-14T18:10:00.933 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:10:00 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-c[155141]: starting mds.c at 2023-07-14T18:10:00.933 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:10:00 smithi187 systemd[1]: Started Ceph mds.c for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:10:01.385 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:10:01.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 -- 172.21.15.115:0/3716121637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 msgr2=0x7f16fc106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:01.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 --2- 172.21.15.115:0/3716121637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 0x7f16fc106f70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f16f400b0a0 tx=0x7f16f400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 -- 172.21.15.115:0/3716121637 shutdown_connections 2023-07-14T18:10:01.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 --2- 172.21.15.115:0/3716121637 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 0x7f16fc106f70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 --2- 172.21.15.115:0/3716121637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc0feb30 0x7f16fc0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 --2- 172.21.15.115:0/3716121637 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16fc0ffd20 0x7f16fc100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 -- 172.21.15.115:0/3716121637 >> 172.21.15.115:0/3716121637 conn(0x7f16fc0fa150 msgr2=0x7f16fc0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:01.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 -- 172.21.15.115:0/3716121637 shutdown_connections 2023-07-14T18:10:01.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 -- 172.21.15.115:0/3716121637 wait complete. 2023-07-14T18:10:01.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 Processor -- start 2023-07-14T18:10:01.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.483+0000 7f17025d8700 1 -- start start 2023-07-14T18:10:01.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f17025d8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16fc0feb30 0x7f16fc069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:01.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f17025d8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc0ffd20 0x7f16fc19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:01.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f17025d8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 0x7f16fc19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:01.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f17025d8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc19f6a0 con 0x7f16fc1006b0 2023-07-14T18:10:01.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f17025d8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc19f810 con 0x7f16fc0ffd20 2023-07-14T18:10:01.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f17025d8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc19f9d0 con 0x7f16fc0feb30 2023-07-14T18:10:01.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f1700b75700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 0x7f16fc19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:01.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f1700b75700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 0x7f16fc19eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52352/0 (socket says 172.21.15.115:52352) 2023-07-14T18:10:01.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f1700b75700 1 -- 172.21.15.115:0/1662271415 learned_addr learned my addr 172.21.15.115:0/1662271415 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:01.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f16fbfff700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16fc0feb30 0x7f16fc069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:01.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f16fb7fe700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc0ffd20 0x7f16fc19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:01.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f1700b75700 1 -- 172.21.15.115:0/1662271415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16fc0feb30 msgr2=0x7f16fc069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:01.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f1700b75700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16fc0feb30 0x7f16fc069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f1700b75700 1 -- 172.21.15.115:0/1662271415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc0ffd20 msgr2=0x7f16fc19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:01.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f1700b75700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc0ffd20 0x7f16fc19caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f1700b75700 1 -- 172.21.15.115:0/1662271415 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f4009d00 con 0x7f16fc1006b0 2023-07-14T18:10:01.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f16fb7fe700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc0ffd20 0x7f16fc19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:01.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f1700b75700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 0x7f16fc19eff0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009fd0 tx=0x7f16f400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:01.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.484+0000 7f16f97fa700 1 -- 172.21.15.115:0/1662271415 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f400e040 con 0x7f16fc1006b0 2023-07-14T18:10:01.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.485+0000 7f16f97fa700 1 -- 172.21.15.115:0/1662271415 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f16f4007760 con 0x7f16fc1006b0 2023-07-14T18:10:01.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.485+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16fc19fc50 con 0x7f16fc1006b0 2023-07-14T18:10:01.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.485+0000 7f16f97fa700 1 -- 172.21.15.115:0/1662271415 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f401b6c0 con 0x7f16fc1006b0 2023-07-14T18:10:01.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.485+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16fc1a0070 con 0x7f16fc1006b0 2023-07-14T18:10:01.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.485+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16fc10a280 con 0x7f16fc1006b0 2023-07-14T18:10:01.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.487+0000 7f16f97fa700 1 -- 172.21.15.115:0/1662271415 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f16f401b820 con 0x7f16fc1006b0 2023-07-14T18:10:01.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.488+0000 7f16f97fa700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16e80a11a0 0x7f16e80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:01.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.488+0000 7f16fbfff700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16e80a11a0 0x7f16e80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:01.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.488+0000 7f16f97fa700 1 -- 172.21.15.115:0/1662271415 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f16f40cdc90 con 0x7f16fc1006b0 2023-07-14T18:10:01.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.488+0000 7f16fbfff700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16e80a11a0 0x7f16e80a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0ffb80 tx=0x7f16e4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:01.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.491+0000 7f16f97fa700 1 -- 172.21.15.115:0/1662271415 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f16f4097910 con 0x7f16fc1006b0 2023-07-14T18:10:01.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.730+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16fc105970 con 0x7f16fc1006b0 2023-07-14T18:10:01.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.730+0000 7f16f97fa700 1 -- 172.21.15.115:0/1662271415 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5814 (secure 0 0 0) 0x7f16f409af30 con 0x7f16fc1006b0 2023-07-14T18:10:01.769 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:01.769 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-14T18:10:01.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16e80a11a0 msgr2=0x7f16e80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:01.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16e80a11a0 0x7f16e80a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0ffb80 tx=0x7f16e4009260 comp rx=0 tx=0).stop 2023-07-14T18:10:01.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 msgr2=0x7f16fc19eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:01.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 0x7f16fc19eff0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009fd0 tx=0x7f16f400bc80 comp rx=0 tx=0).stop 2023-07-14T18:10:01.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 shutdown_connections 2023-07-14T18:10:01.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16e80a11a0 0x7f16e80a3650 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f16fc1006b0 0x7f16fc19eff0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16fc0feb30 0x7f16fc069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 --2- 172.21.15.115:0/1662271415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc0ffd20 0x7f16fc19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:01.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 >> 172.21.15.115:0/1662271415 conn(0x7f16fc0fa150 msgr2=0x7f16fc0fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:01.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 shutdown_connections 2023-07-14T18:10:01.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:01.733+0000 7f17025d8700 1 -- 172.21.15.115:0/1662271415 wait complete. 2023-07-14T18:10:01.773 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 8 2023-07-14T18:10:01.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:01.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:01.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:01.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:01.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:01.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:01.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: Deploying daemon mds.h on smithi150 2023-07-14T18:10:01.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:01.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.187:6836/771796341,v1:172.21.15.187:6837/771796341] up:boot 2023-07-14T18:10:01.783 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: fsmap 7 up:standby 2023-07-14T18:10:01.784 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:01 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:10:01.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:01.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:01.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:01.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:01.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:01.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:01.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: Deploying daemon mds.h on smithi150 2023-07-14T18:10:01.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:01.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.187:6836/771796341,v1:172.21.15.187:6837/771796341] up:boot 2023-07-14T18:10:01.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: fsmap 7 up:standby 2023-07-14T18:10:01.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:01 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:10:01.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:01.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:01.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:01.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:01.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:01.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:01.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: Deploying daemon mds.h on smithi150 2023-07-14T18:10:01.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:01.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.187:6836/771796341,v1:172.21.15.187:6837/771796341] up:boot 2023-07-14T18:10:01.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: fsmap 7 up:standby 2023-07-14T18:10:01.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:01 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:10:02.336 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:10:02 smithi150 systemd[1]: Starting Ceph mds.h for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:10:02.620 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:10:02 smithi150 podman[153666]: 2023-07-14T18:10:02.620 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:10:02 smithi150 bash[153666]: 8f54d8cfb00cebe49655b1d2c260f69cf5857d093cc7b72c940ac2d23f866ab3 2023-07-14T18:10:02.620 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:10:02 smithi150 ceph-mds[153707]: -- 172.21.15.150:0/3701847369 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5564940911e0 con 0x556494066c00 2023-07-14T18:10:02.621 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:10:02 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-h[153682]: starting mds.h at 2023-07-14T18:10:02.932 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:02 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1662271415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:03.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:02 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1662271415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:03.120 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:10:02 smithi150 systemd[1]: Started Ceph mds.h for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:10:03.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:02 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1662271415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:03.785 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:10:03.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 -- 172.21.15.115:0/3682678300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c103000 msgr2=0x7f656c1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:03.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 --2- 172.21.15.115:0/3682678300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c103000 0x7f656c1053e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6554009990 tx=0x7f6554009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:03.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 -- 172.21.15.115:0/3682678300 shutdown_connections 2023-07-14T18:10:03.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 --2- 172.21.15.115:0/3682678300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f656c105920 0x7f656c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:03.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 --2- 172.21.15.115:0/3682678300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f656c1006e0 0x7f656c102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:03.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 --2- 172.21.15.115:0/3682678300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c103000 0x7f656c1053e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:03.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 -- 172.21.15.115:0/3682678300 >> 172.21.15.115:0/3682678300 conn(0x7f656c0fa100 msgr2=0x7f656c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:03.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 -- 172.21.15.115:0/3682678300 shutdown_connections 2023-07-14T18:10:03.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 -- 172.21.15.115:0/3682678300 wait complete. 2023-07-14T18:10:03.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 Processor -- start 2023-07-14T18:10:03.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.882+0000 7f6571d7a700 1 -- start start 2023-07-14T18:10:03.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6571d7a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c1006e0 0x7f656c100050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:03.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6571d7a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f656c103000 0x7f656c0fe690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:03.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6571d7a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f656c105920 0x7f656c0febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:03.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6571d7a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c0ff370 con 0x7f656c1006e0 2023-07-14T18:10:03.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6571d7a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c0ff4e0 con 0x7f656c103000 2023-07-14T18:10:03.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6571d7a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c19f740 con 0x7f656c105920 2023-07-14T18:10:03.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c1006e0 0x7f656c100050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:03.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c1006e0 0x7f656c100050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52362/0 (socket says 172.21.15.115:52362) 2023-07-14T18:10:03.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656b7fe700 1 -- 172.21.15.115:0/215819305 learned_addr learned my addr 172.21.15.115:0/215819305 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:03.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656bfff700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f656c105920 0x7f656c0febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:03.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656affd700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f656c103000 0x7f656c0fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:03.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656b7fe700 1 -- 172.21.15.115:0/215819305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f656c105920 msgr2=0x7f656c0febd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:03.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656b7fe700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f656c105920 0x7f656c0febd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:03.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656b7fe700 1 -- 172.21.15.115:0/215819305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f656c103000 msgr2=0x7f656c0fe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:03.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656b7fe700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f656c103000 0x7f656c0fe690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:03.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656b7fe700 1 -- 172.21.15.115:0/215819305 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65540095f0 con 0x7f656c1006e0 2023-07-14T18:10:03.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f656b7fe700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c1006e0 0x7f656c100050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6564007e20 tx=0x7f656400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:03.925 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6568ff9700 1 -- 172.21.15.115:0/215819305 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656400ad40 con 0x7f656c1006e0 2023-07-14T18:10:03.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c19fa20 con 0x7f656c1006e0 2023-07-14T18:10:03.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6568ff9700 1 -- 172.21.15.115:0/215819305 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f65640128b0 con 0x7f656c1006e0 2023-07-14T18:10:03.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6568ff9700 1 -- 172.21.15.115:0/215819305 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65640116b0 con 0x7f656c1006e0 2023-07-14T18:10:03.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.883+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c19ff30 con 0x7f656c1006e0 2023-07-14T18:10:03.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.884+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c0fbc70 con 0x7f656c1006e0 2023-07-14T18:10:03.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.885+0000 7f6568ff9700 1 -- 172.21.15.115:0/215819305 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f6564012a20 con 0x7f656c1006e0 2023-07-14T18:10:03.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.886+0000 7f6568ff9700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f65580a11a0 0x7f65580a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:03.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.886+0000 7f656affd700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f65580a11a0 0x7f65580a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:03.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.886+0000 7f6568ff9700 1 -- 172.21.15.115:0/215819305 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f65640cc850 con 0x7f656c1006e0 2023-07-14T18:10:03.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.886+0000 7f656affd700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f65580a11a0 0x7f65580a3650 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6554006010 tx=0x7f6554000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:03.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:03.889+0000 7f6568ff9700 1 -- 172.21.15.115:0/215819305 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6564096440 con 0x7f656c1006e0 2023-07-14T18:10:03.956 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:03.956 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:03.956 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:03.957 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:03.957 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:03.957 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:03.957 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: Deploying daemon mds.i on smithi187 2023-07-14T18:10:03.957 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:03.957 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.150:6836/1462100932,v1:172.21.15.150:6837/1462100932] up:boot 2023-07-14T18:10:03.957 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: fsmap 8 up:standby 2023-07-14T18:10:03.958 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:03 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:10:04.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:04.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:04.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:04.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:04.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:04.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:04.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: Deploying daemon mds.i on smithi187 2023-07-14T18:10:04.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:04.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.150:6836/1462100932,v1:172.21.15.150:6837/1462100932] up:boot 2023-07-14T18:10:04.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: fsmap 8 up:standby 2023-07-14T18:10:04.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:03 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:10:04.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:04.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:04.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:04.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:04.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:04.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:04.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: Deploying daemon mds.i on smithi187 2023-07-14T18:10:04.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:04.121 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.150:6836/1462100932,v1:172.21.15.150:6837/1462100932] up:boot 2023-07-14T18:10:04.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: fsmap 8 up:standby 2023-07-14T18:10:04.122 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:03 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:10:04.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.123+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f656c02c8e0 con 0x7f656c1006e0 2023-07-14T18:10:04.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.123+0000 7f6568ff9700 1 -- 172.21.15.115:0/215819305 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6574 (secure 0 0 0) 0x7f6564099a60 con 0x7f656c1006e0 2023-07-14T18:10:04.162 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:04.162 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T18:10:04.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.125+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f65580a11a0 msgr2=0x7f65580a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:04.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.125+0000 7f6571d7a700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f65580a11a0 0x7f65580a3650 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6554006010 tx=0x7f6554000bc0 comp rx=0 tx=0).stop 2023-07-14T18:10:04.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.125+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c1006e0 msgr2=0x7f656c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:04.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.125+0000 7f6571d7a700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c1006e0 0x7f656c100050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6564007e20 tx=0x7f656400a340 comp rx=0 tx=0).stop 2023-07-14T18:10:04.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.125+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 shutdown_connections 2023-07-14T18:10:04.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.125+0000 7f6571d7a700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f65580a11a0 0x7f65580a3650 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:04.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.125+0000 7f6571d7a700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f656c105920 0x7f656c0febd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:04.164 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.125+0000 7f6571d7a700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f656c1006e0 0x7f656c100050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:04.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.125+0000 7f6571d7a700 1 --2- 172.21.15.115:0/215819305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f656c103000 0x7f656c0fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:04.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.126+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 >> 172.21.15.115:0/215819305 conn(0x7f656c0fa100 msgr2=0x7f656c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:04.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.126+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 shutdown_connections 2023-07-14T18:10:04.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:04.126+0000 7f6571d7a700 1 -- 172.21.15.115:0/215819305 wait complete. 2023-07-14T18:10:04.165 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 9 2023-07-14T18:10:04.291 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:10:04 smithi187 systemd[1]: Starting Ceph mds.i for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:10:04.813 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:10:04 smithi187 podman[155466]: 2023-07-14T18:10:04.814 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:04 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/215819305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:05.104 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:04 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/215819305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:05.120 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:04 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/215819305' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:05.183 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:10:04 smithi187 ceph-mds[155506]: -- 172.21.15.187:0/1999778172 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dfa05651e0 con 0x55dfa053b000 2023-07-14T18:10:05.183 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:10:04 smithi187 bash[155466]: f2068be83577c989084f2f25116f683b04772a0adfb6677d88cd00ea554806f2 2023-07-14T18:10:05.183 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:10:04 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-i[155482]: starting mds.i at 2023-07-14T18:10:05.184 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:10:04 smithi187 systemd[1]: Started Ceph mds.i for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:10:06.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:10:06.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:06.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:06.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:06.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:06.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:06.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:06.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:06.183 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: Deploying daemon mds.g on smithi115 2023-07-14T18:10:06.184 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] up:boot 2023-07-14T18:10:06.184 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: fsmap 9 up:standby 2023-07-14T18:10:06.184 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:05 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:10:06.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:06.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:06.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:06.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:06.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:06.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:06.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:06.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: Deploying daemon mds.g on smithi115 2023-07-14T18:10:06.204 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] up:boot 2023-07-14T18:10:06.205 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: fsmap 9 up:standby 2023-07-14T18:10:06.205 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:05 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:10:06.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.267+0000 7fc2aca62700 1 -- 172.21.15.115:0/1215998798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80ffcf0 msgr2=0x7fc2a8100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:06.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.267+0000 7fc2aca62700 1 --2- 172.21.15.115:0/1215998798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80ffcf0 0x7fc2a8100140 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc298009990 tx=0x7fc298009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.267+0000 7fc2aca62700 1 -- 172.21.15.115:0/1215998798 shutdown_connections 2023-07-14T18:10:06.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.267+0000 7fc2aca62700 1 --2- 172.21.15.115:0/1215998798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a8100680 0x7fc2a8106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.267+0000 7fc2aca62700 1 --2- 172.21.15.115:0/1215998798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2a80feb00 0x7fc2a80fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.267+0000 7fc2aca62700 1 --2- 172.21.15.115:0/1215998798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80ffcf0 0x7fc2a8100140 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.267+0000 7fc2aca62700 1 -- 172.21.15.115:0/1215998798 >> 172.21.15.115:0/1215998798 conn(0x7fc2a80fa100 msgr2=0x7fc2a80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:06.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.267+0000 7fc2aca62700 1 -- 172.21.15.115:0/1215998798 shutdown_connections 2023-07-14T18:10:06.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.267+0000 7fc2aca62700 1 -- 172.21.15.115:0/1215998798 wait complete. 2023-07-14T18:10:06.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2aca62700 1 Processor -- start 2023-07-14T18:10:06.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2aca62700 1 -- start start 2023-07-14T18:10:06.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2aca62700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80feb00 0x7fc2a8102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:06.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2aca62700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80ffcf0 0x7fc2a81010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:06.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2aca62700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2a8100680 0x7fc2a8101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:06.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2aca62700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8101db0 con 0x7fc2a80feb00 2023-07-14T18:10:06.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2aca62700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8101f20 con 0x7fc2a80ffcf0 2023-07-14T18:10:06.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2aca62700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a819f8b0 con 0x7fc2a8100680 2023-07-14T18:10:06.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2a659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80feb00 0x7fc2a8102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:06.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.268+0000 7fc2a659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80feb00 0x7fc2a8102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52376/0 (socket says 172.21.15.115:52376) 2023-07-14T18:10:06.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2a659c700 1 -- 172.21.15.115:0/2614525472 learned_addr learned my addr 172.21.15.115:0/2614525472 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:06.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2a6d9d700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2a8100680 0x7fc2a8101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:06.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2a5d9b700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80ffcf0 0x7fc2a81010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:06.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2a659c700 1 -- 172.21.15.115:0/2614525472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2a8100680 msgr2=0x7fc2a8101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:06.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2a659c700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2a8100680 0x7fc2a8101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2a659c700 1 -- 172.21.15.115:0/2614525472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80ffcf0 msgr2=0x7fc2a81010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:06.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2a659c700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80ffcf0 0x7fc2a81010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2a659c700 1 -- 172.21.15.115:0/2614525472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2980095f0 con 0x7fc2a80feb00 2023-07-14T18:10:06.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2a659c700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80feb00 0x7fc2a8102a90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc29000d980 tx=0x7fc29000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:06.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc29f7fe700 1 -- 172.21.15.115:0/2614525472 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000f890 con 0x7fc2a80feb00 2023-07-14T18:10:06.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc29f7fe700 1 -- 172.21.15.115:0/2614525472 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc29000fe80 con 0x7fc2a80feb00 2023-07-14T18:10:06.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a819fb90 con 0x7fc2a80feb00 2023-07-14T18:10:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc29f7fe700 1 -- 172.21.15.115:0/2614525472 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000e3b0 con 0x7fc2a80feb00 2023-07-14T18:10:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.269+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a81a00a0 con 0x7fc2a80feb00 2023-07-14T18:10:06.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.270+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a810a1a0 con 0x7fc2a80feb00 2023-07-14T18:10:06.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.271+0000 7fc29f7fe700 1 -- 172.21.15.115:0/2614525472 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fc29000f9f0 con 0x7fc2a80feb00 2023-07-14T18:10:06.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.272+0000 7fc29f7fe700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc2940a1220 0x7fc2940a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:06.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.272+0000 7fc29f7fe700 1 -- 172.21.15.115:0/2614525472 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc2900cbea0 con 0x7fc2a80feb00 2023-07-14T18:10:06.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.272+0000 7fc2a5d9b700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc2940a1220 0x7fc2940a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:06.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.272+0000 7fc2a5d9b700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc2940a1220 0x7fc2940a36d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc29800c010 tx=0x7fc298005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:06.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.275+0000 7fc29f7fe700 1 -- 172.21.15.115:0/2614525472 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc290095b10 con 0x7fc2a80feb00 2023-07-14T18:10:06.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:06.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:06.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:06.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:06.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:06.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:06.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:06.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: Deploying daemon mds.g on smithi115 2023-07-14T18:10:06.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] up:boot 2023-07-14T18:10:06.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: fsmap 9 up:standby 2023-07-14T18:10:06.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:05 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:10:06.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.536+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2a810a390 con 0x7fc2a80feb00 2023-07-14T18:10:06.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.538+0000 7fc29f7fe700 1 -- 172.21.15.115:0/2614525472 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7336 (secure 0 0 0) 0x7fc290099130 con 0x7fc2a80feb00 2023-07-14T18:10:06.576 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:06.576 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T18:10:06.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc2940a1220 msgr2=0x7fc2940a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:06.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc2940a1220 0x7fc2940a36d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc29800c010 tx=0x7fc298005960 comp rx=0 tx=0).stop 2023-07-14T18:10:06.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80feb00 msgr2=0x7fc2a8102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:06.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80feb00 0x7fc2a8102a90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc29000d980 tx=0x7fc29000dc90 comp rx=0 tx=0).stop 2023-07-14T18:10:06.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 shutdown_connections 2023-07-14T18:10:06.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc2940a1220 0x7fc2940a36d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc2a8100680 0x7fc2a8101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2a80feb00 0x7fc2a8102a90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 --2- 172.21.15.115:0/2614525472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2a80ffcf0 0x7fc2a81010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:06.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 >> 172.21.15.115:0/2614525472 conn(0x7fc2a80fa100 msgr2=0x7fc2a80677a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:06.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 shutdown_connections 2023-07-14T18:10:06.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:06.540+0000 7fc2aca62700 1 -- 172.21.15.115:0/2614525472 wait complete. 2023-07-14T18:10:06.580 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 10 2023-07-14T18:10:06.855 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:10:06 smithi115 systemd[1]: Starting Ceph mds.g for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:10:07.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:06 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2614525472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:07.108 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:10:06 smithi115 podman[161774]: 2023-07-14T18:10:07.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:06 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2614525472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:07.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:06 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2614525472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:07.605 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:10:07 smithi115 ceph-mds[161815]: -- 172.21.15.115:0/4030679758 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fc0ab351e0 con 0x55fc0ab0a800 2023-07-14T18:10:07.605 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:10:07 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-g[161790]: starting mds.g at 2023-07-14T18:10:07.605 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:10:07 smithi115 bash[161774]: a7f80b32166eafbbbf4c86b0c57f1b71749422fb78c0404a8932704875c44049 2023-07-14T18:10:07.605 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:10:07 smithi115 systemd[1]: Started Ceph mds.g for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:10:08.426 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:08.426 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:08.426 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:08.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:08.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:08.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:08.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:08.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: Deploying daemon mds.a on smithi115 2023-07-14T18:10:08.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.115:6838/2547274817,v1:172.21.15.115:6839/2547274817] up:boot 2023-07-14T18:10:08.427 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: fsmap 10 up:standby 2023-07-14T18:10:08.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:08 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T18:10:08.519 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:08.520 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:08.520 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:08.520 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:08.521 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:08.521 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:08.521 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:08.521 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: Deploying daemon mds.a on smithi115 2023-07-14T18:10:08.522 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.115:6838/2547274817,v1:172.21.15.115:6839/2547274817] up:boot 2023-07-14T18:10:08.522 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: fsmap 10 up:standby 2023-07-14T18:10:08.522 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:08 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T18:10:08.592 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:10:08.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:08.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:08.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:08.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:08.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:08.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:08.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:08.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: Deploying daemon mds.a on smithi115 2023-07-14T18:10:08.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.115:6838/2547274817,v1:172.21.15.115:6839/2547274817] up:boot 2023-07-14T18:10:08.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: fsmap 10 up:standby 2023-07-14T18:10:08.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:08 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T18:10:08.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.716+0000 7f414c6b7700 1 -- 172.21.15.115:0/1958959520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41441000d0 msgr2=0x7f41441004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:08.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.716+0000 7f414c6b7700 1 --2- 172.21.15.115:0/1958959520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41441000d0 0x7f41441004e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4140009990 tx=0x7f4140009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:08.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.716+0000 7f414c6b7700 1 -- 172.21.15.115:0/1958959520 shutdown_connections 2023-07-14T18:10:08.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.716+0000 7f414c6b7700 1 --2- 172.21.15.115:0/1958959520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41440ff6d0 0x7f4144106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:08.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.716+0000 7f414c6b7700 1 --2- 172.21.15.115:0/1958959520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41441000d0 0x7f41441004e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:08.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.716+0000 7f414c6b7700 1 --2- 172.21.15.115:0/1958959520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41440fed20 0x7f41440ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:08.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.716+0000 7f414c6b7700 1 -- 172.21.15.115:0/1958959520 >> 172.21.15.115:0/1958959520 conn(0x7f41440f98a0 msgr2=0x7f41440fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:08.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.716+0000 7f414c6b7700 1 -- 172.21.15.115:0/1958959520 shutdown_connections 2023-07-14T18:10:08.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.716+0000 7f414c6b7700 1 -- 172.21.15.115:0/1958959520 wait complete. 2023-07-14T18:10:08.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414c6b7700 1 Processor -- start 2023-07-14T18:10:08.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414c6b7700 1 -- start start 2023-07-14T18:10:08.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414c6b7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41440fed20 0x7f4144195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414c6b7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41440ff6d0 0x7f4144195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414c6b7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41441000d0 0x7f414419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414c6b7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414419d3f0 con 0x7f41440ff6d0 2023-07-14T18:10:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414c6b7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414419d530 con 0x7f41441000d0 2023-07-14T18:10:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414c6b7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414419d6c0 con 0x7f41440fed20 2023-07-14T18:10:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f4149c52700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41440ff6d0 0x7f4144195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f4149c52700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41440ff6d0 0x7f4144195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52402/0 (socket says 172.21.15.115:52402) 2023-07-14T18:10:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f4149c52700 1 -- 172.21.15.115:0/3218221208 learned_addr learned my addr 172.21.15.115:0/3218221208 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414a453700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41440fed20 0x7f4144195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414ac54700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41441000d0 0x7f414419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f4149c52700 1 -- 172.21.15.115:0/3218221208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41440fed20 msgr2=0x7f4144195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f4149c52700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41440fed20 0x7f4144195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f4149c52700 1 -- 172.21.15.115:0/3218221208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41441000d0 msgr2=0x7f414419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:08.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f4149c52700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41441000d0 0x7f414419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:08.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f4149c52700 1 -- 172.21.15.115:0/3218221208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41400095f0 con 0x7f41440ff6d0 2023-07-14T18:10:08.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f414ac54700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41441000d0 0x7f414419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:08.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f4149c52700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41440ff6d0 0x7f4144195db0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f413400d980 tx=0x7f413400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:08.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.717+0000 7f413b7fe700 1 -- 172.21.15.115:0/3218221208 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4134004020 con 0x7f41440ff6d0 2023-07-14T18:10:08.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.718+0000 7f413b7fe700 1 -- 172.21.15.115:0/3218221208 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4134009730 con 0x7f41440ff6d0 2023-07-14T18:10:08.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.718+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414419d9a0 con 0x7f41440ff6d0 2023-07-14T18:10:08.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.718+0000 7f413b7fe700 1 -- 172.21.15.115:0/3218221208 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4134003a00 con 0x7f41440ff6d0 2023-07-14T18:10:08.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.718+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414419df00 con 0x7f41440ff6d0 2023-07-14T18:10:08.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.718+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4144106cf0 con 0x7f41440ff6d0 2023-07-14T18:10:08.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.720+0000 7f413b7fe700 1 -- 172.21.15.115:0/3218221208 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f41340098a0 con 0x7f41440ff6d0 2023-07-14T18:10:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.720+0000 7f413b7fe700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41300a1220 0x7f41300a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.721+0000 7f413b7fe700 1 -- 172.21.15.115:0/3218221208 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f41340cc410 con 0x7f41440ff6d0 2023-07-14T18:10:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.721+0000 7f414a453700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41300a1220 0x7f41300a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.721+0000 7f414a453700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41300a1220 0x7f41300a36d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f414000b5c0 tx=0x7f4140005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.723+0000 7f413b7fe700 1 -- 172.21.15.115:0/3218221208 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4134096080 con 0x7f41440ff6d0 2023-07-14T18:10:09.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.982+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f414402c8e0 con 0x7f41440ff6d0 2023-07-14T18:10:09.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.982+0000 7f413b7fe700 1 -- 172.21.15.115:0/3218221208 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8097 (secure 0 0 0) 0x7f41340996a0 con 0x7f41440ff6d0 2023-07-14T18:10:09.021 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:09.022 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T18:10:09.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41300a1220 msgr2=0x7f41300a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:09.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41300a1220 0x7f41300a36d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f414000b5c0 tx=0x7f4140005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:09.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41440ff6d0 msgr2=0x7f4144195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:09.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41440ff6d0 0x7f4144195db0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f413400d980 tx=0x7f413400dc90 comp rx=0 tx=0).stop 2023-07-14T18:10:09.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 shutdown_connections 2023-07-14T18:10:09.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f41300a1220 0x7f41300a36d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:09.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41441000d0 0x7f414419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:09.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41440fed20 0x7f4144195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:09.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 --2- 172.21.15.115:0/3218221208 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41440ff6d0 0x7f4144195db0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:09.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 >> 172.21.15.115:0/3218221208 conn(0x7f41440f98a0 msgr2=0x7f41440fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:09.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 shutdown_connections 2023-07-14T18:10:09.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:08.984+0000 7f414c6b7700 1 -- 172.21.15.115:0/3218221208 wait complete. 2023-07-14T18:10:09.026 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 11 2023-07-14T18:10:09.034 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:10:08 smithi115 systemd[1]: Starting Ceph mds.a for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:10:09.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:09 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3218221208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:09.313 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:10:09 smithi115 podman[162160]: 2023-07-14T18:10:09.605 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:10:09 smithi115 ceph-mds[162202]: -- 172.21.15.115:0/2594558643 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5600c9da71e0 con 0x5600c9d7c800 2023-07-14T18:10:09.605 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:10:09 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-a[162178]: starting mds.a at 2023-07-14T18:10:09.605 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:10:09 smithi115 bash[162160]: 85d35ede4f289250b79ff2bf51b242acf5bd4a8d85b2127399510c65dedcd50d 2023-07-14T18:10:09.605 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:10:09 smithi115 systemd[1]: Started Ceph mds.a for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:10:09.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:09 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3218221208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:09.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:09 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3218221208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:10.285 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:10 smithi150 ceph-mon[128853]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:10.285 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:10.286 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:10.286 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:10.286 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:10.286 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:10.286 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:10 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:10.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:10 smithi115 ceph-mon[127451]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:10.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:10.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:10.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:10.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:10.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:10.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:10 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:10.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:10 smithi187 ceph-mon[129317]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:10.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:10.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:10.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:10.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:10:10.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:10:10.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:10 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:11.035 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:10:11.120 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:10:10 smithi150 systemd[1]: Starting Ceph mds.b for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:10:11.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.135+0000 7f7b74d73700 1 -- 172.21.15.115:0/142516971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b701017d0 msgr2=0x7f7b70108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:11.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.135+0000 7f7b74d73700 1 --2- 172.21.15.115:0/142516971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b701017d0 0x7f7b70108090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800b0a0 tx=0x7f7b6800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.135+0000 7f7b74d73700 1 -- 172.21.15.115:0/142516971 shutdown_connections 2023-07-14T18:10:11.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.135+0000 7f7b74d73700 1 --2- 172.21.15.115:0/142516971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b701017d0 0x7f7b70108090 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.135+0000 7f7b74d73700 1 --2- 172.21.15.115:0/142516971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b700ffc50 0x7f7b70100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.135+0000 7f7b74d73700 1 --2- 172.21.15.115:0/142516971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b70100e40 0x7f7b70101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.135+0000 7f7b74d73700 1 -- 172.21.15.115:0/142516971 >> 172.21.15.115:0/142516971 conn(0x7f7b700fb250 msgr2=0x7f7b700fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:11.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.135+0000 7f7b74d73700 1 -- 172.21.15.115:0/142516971 shutdown_connections 2023-07-14T18:10:11.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.135+0000 7f7b74d73700 1 -- 172.21.15.115:0/142516971 wait complete. 2023-07-14T18:10:11.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b74d73700 1 Processor -- start 2023-07-14T18:10:11.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b74d73700 1 -- start start 2023-07-14T18:10:11.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b74d73700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b700ffc50 0x7f7b70195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:11.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b74d73700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b70100e40 0x7f7b70195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:11.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b74d73700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b701017d0 0x7f7b7019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:11.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b74d73700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7019d480 con 0x7f7b70100e40 2023-07-14T18:10:11.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b74d73700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7019d5c0 con 0x7f7b701017d0 2023-07-14T18:10:11.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b74d73700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7019d750 con 0x7f7b700ffc50 2023-07-14T18:10:11.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b70100e40 0x7f7b70195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:11.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b70100e40 0x7f7b70195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52428/0 (socket says 172.21.15.115:52428) 2023-07-14T18:10:11.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6ed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b701017d0 0x7f7b7019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:11.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6dd9b700 1 -- 172.21.15.115:0/75397904 learned_addr learned my addr 172.21.15.115:0/75397904 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:11.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6e59c700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b700ffc50 0x7f7b70195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:11.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6ed9d700 1 -- 172.21.15.115:0/75397904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b700ffc50 msgr2=0x7f7b70195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:11.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6ed9d700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b700ffc50 0x7f7b70195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6ed9d700 1 -- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b70100e40 msgr2=0x7f7b70195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:11.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6ed9d700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b70100e40 0x7f7b70195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.178 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6ed9d700 1 -- 172.21.15.115:0/75397904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b68009d00 con 0x7f7b701017d0 2023-07-14T18:10:11.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6dd9b700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b70100e40 0x7f7b70195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:10:11.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6e59c700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b700ffc50 0x7f7b70195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:11.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.136+0000 7f7b6ed9d700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b701017d0 0x7f7b7019ceb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7b68015000 tx=0x7f7b6800ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:11.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.137+0000 7f7b677fe700 1 -- 172.21.15.115:0/75397904 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6800e040 con 0x7f7b701017d0 2023-07-14T18:10:11.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.137+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7019d9d0 con 0x7f7b701017d0 2023-07-14T18:10:11.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.137+0000 7f7b677fe700 1 -- 172.21.15.115:0/75397904 <== mon.2 v2:172.21.15.150:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7b68003c00 con 0x7f7b701017d0 2023-07-14T18:10:11.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.137+0000 7f7b677fe700 1 -- 172.21.15.115:0/75397904 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b680052a0 con 0x7f7b701017d0 2023-07-14T18:10:11.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.137+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7019dea0 con 0x7f7b701017d0 2023-07-14T18:10:11.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.138+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7010b2e0 con 0x7f7b701017d0 2023-07-14T18:10:11.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.139+0000 7f7b677fe700 1 -- 172.21.15.115:0/75397904 <== mon.2 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7b6801d470 con 0x7f7b701017d0 2023-07-14T18:10:11.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.139+0000 7f7b677fe700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b5c0a11a0 0x7f7b5c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:11.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.139+0000 7f7b6e59c700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b5c0a11a0 0x7f7b5c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:11.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.139+0000 7f7b677fe700 1 -- 172.21.15.115:0/75397904 <== mon.2 v2:172.21.15.150:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7b680cd8c0 con 0x7f7b701017d0 2023-07-14T18:10:11.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.140+0000 7f7b6e59c700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b5c0a11a0 0x7f7b5c0a3650 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7b58005950 tx=0x7f7b580058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:11.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.143+0000 7f7b677fe700 1 -- 172.21.15.115:0/75397904 <== mon.2 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7b68097540 con 0x7f7b701017d0 2023-07-14T18:10:11.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:11 smithi115 ceph-mon[127451]: Deploying daemon mds.b on smithi150 2023-07-14T18:10:11.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:11 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.115:6840/2429034282,v1:172.21.15.115:6841/2429034282] up:boot 2023-07-14T18:10:11.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:11 smithi115 ceph-mon[127451]: fsmap 11 up:standby 2023-07-14T18:10:11.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:11 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T18:10:11.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.378+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b70106ad0 con 0x7f7b701017d0 2023-07-14T18:10:11.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.379+0000 7f7b677fe700 1 -- 172.21.15.115:0/75397904 <== mon.2 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8858 (secure 0 0 0) 0x7f7b6809ab60 con 0x7f7b701017d0 2023-07-14T18:10:11.417 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:11.417 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T18:10:11.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b5c0a11a0 msgr2=0x7f7b5c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:11.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b5c0a11a0 0x7f7b5c0a3650 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7b58005950 tx=0x7f7b580058e0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b701017d0 msgr2=0x7f7b7019ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:11.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b701017d0 0x7f7b7019ceb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7b68015000 tx=0x7f7b6800ba50 comp rx=0 tx=0).stop 2023-07-14T18:10:11.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 shutdown_connections 2023-07-14T18:10:11.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7b5c0a11a0 0x7f7b5c0a3650 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b701017d0 0x7f7b7019ceb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b700ffc50 0x7f7b70195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 --2- 172.21.15.115:0/75397904 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7b70100e40 0x7f7b70195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:11.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 >> 172.21.15.115:0/75397904 conn(0x7f7b700fb250 msgr2=0x7f7b700fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:11.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 shutdown_connections 2023-07-14T18:10:11.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:11.381+0000 7f7b74d73700 1 -- 172.21.15.115:0/75397904 wait complete. 2023-07-14T18:10:11.421 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 12 2023-07-14T18:10:11.592 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:10:11 smithi150 podman[154121]: 2023-07-14T18:10:11.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:11 smithi150 ceph-mon[128853]: Deploying daemon mds.b on smithi150 2023-07-14T18:10:11.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:11 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.115:6840/2429034282,v1:172.21.15.115:6841/2429034282] up:boot 2023-07-14T18:10:11.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:11 smithi150 ceph-mon[128853]: fsmap 11 up:standby 2023-07-14T18:10:11.593 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:11 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T18:10:11.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:11 smithi187 ceph-mon[129317]: Deploying daemon mds.b on smithi150 2023-07-14T18:10:11.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:11 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.115:6840/2429034282,v1:172.21.15.115:6841/2429034282] up:boot 2023-07-14T18:10:11.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:11 smithi187 ceph-mon[129317]: fsmap 11 up:standby 2023-07-14T18:10:11.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:11 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T18:10:11.870 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:10:11 smithi150 ceph-mds[154161]: -- 172.21.15.150:0/3093896627 <== mon.2 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fcaf85f1e0 con 0x55fcaf835000 2023-07-14T18:10:11.870 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:10:11 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-b[154137]: starting mds.b at 2023-07-14T18:10:11.870 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:10:11 smithi150 bash[154121]: e0dc4bcaca7940f1b5fb95ff30451cb77dc4840d109af8f08cb920a33c114bf7 2023-07-14T18:10:11.871 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:10:11 smithi150 systemd[1]: Started Ceph mds.b for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:10:12.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:12 smithi115 ceph-mon[127451]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:12.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:12 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/75397904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:12.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:12 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:10:12.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:12 smithi150 ceph-mon[128853]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:12.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:12 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/75397904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:12.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:12 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:10:12.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:12 smithi187 ceph-mon[129317]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:12.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:12 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/75397904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:12.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:12.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:12 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:10:13.431 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:10:13.469 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:13 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] up:boot 2023-07-14T18:10:13.470 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:13 smithi150 ceph-mon[128853]: fsmap 12 up:standby 2023-07-14T18:10:13.470 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:13 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:10:13.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.524+0000 7fed14281700 1 -- 172.21.15.115:0/263404263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 msgr2=0x7fed0c0ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:13.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.524+0000 7fed14281700 1 --2- 172.21.15.115:0/263404263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 0x7fed0c0ff210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009990 tx=0x7fecfc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.524+0000 7fed14281700 1 -- 172.21.15.115:0/263404263 shutdown_connections 2023-07-14T18:10:13.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.524+0000 7fed14281700 1 --2- 172.21.15.115:0/263404263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed0c0ff750 0x7fed0c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.524+0000 7fed14281700 1 --2- 172.21.15.115:0/263404263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c100100 0x7fed0c100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.524+0000 7fed14281700 1 --2- 172.21.15.115:0/263404263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 0x7fed0c0ff210 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.524+0000 7fed14281700 1 -- 172.21.15.115:0/263404263 >> 172.21.15.115:0/263404263 conn(0x7fed0c0f98a0 msgr2=0x7fed0c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:13.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.524+0000 7fed14281700 1 -- 172.21.15.115:0/263404263 shutdown_connections 2023-07-14T18:10:13.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.524+0000 7fed14281700 1 -- 172.21.15.115:0/263404263 wait complete. 2023-07-14T18:10:13.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed14281700 1 Processor -- start 2023-07-14T18:10:13.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed14281700 1 -- start start 2023-07-14T18:10:13.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed14281700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 0x7fed0c0fe860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:13.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed14281700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed0c0ff750 0x7fed0c19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:13.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed14281700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c100100 0x7fed0c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:13.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed14281700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c19d3f0 con 0x7fed0c0feda0 2023-07-14T18:10:13.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed14281700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c19d560 con 0x7fed0c0ff750 2023-07-14T18:10:13.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed14281700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c19d720 con 0x7fed0c100100 2023-07-14T18:10:13.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1201d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 0x7fed0c0fe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:13.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1201d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 0x7fed0c0fe860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52438/0 (socket says 172.21.15.115:52438) 2023-07-14T18:10:13.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1201d700 1 -- 172.21.15.115:0/2552909414 learned_addr learned my addr 172.21.15.115:0/2552909414 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:13.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1181c700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed0c0ff750 0x7fed0c19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:13.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1201d700 1 -- 172.21.15.115:0/2552909414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c100100 msgr2=0x7fed0c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:13.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1281e700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c100100 0x7fed0c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:13.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1201d700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c100100 0x7fed0c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1201d700 1 -- 172.21.15.115:0/2552909414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed0c0ff750 msgr2=0x7fed0c19a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:13.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1201d700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed0c0ff750 0x7fed0c19a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1201d700 1 -- 172.21.15.115:0/2552909414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0095f0 con 0x7fed0c0feda0 2023-07-14T18:10:13.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1281e700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c100100 0x7fed0c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:13.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1181c700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed0c0ff750 0x7fed0c19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:13.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.525+0000 7fed1201d700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 0x7fed0c0fe860 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fed0800b500 tx=0x7fed0800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:13.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.526+0000 7fed037fe700 1 -- 172.21.15.115:0/2552909414 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08011830 con 0x7fed0c0feda0 2023-07-14T18:10:13.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.526+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c19da00 con 0x7fed0c0feda0 2023-07-14T18:10:13.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.526+0000 7fed037fe700 1 -- 172.21.15.115:0/2552909414 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed08011e20 con 0x7fed0c0feda0 2023-07-14T18:10:13.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.526+0000 7fed037fe700 1 -- 172.21.15.115:0/2552909414 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08010dd0 con 0x7fed0c0feda0 2023-07-14T18:10:13.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.526+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c19df10 con 0x7fed0c0feda0 2023-07-14T18:10:13.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.526+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c10a240 con 0x7fed0c0feda0 2023-07-14T18:10:13.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.527+0000 7fed037fe700 1 -- 172.21.15.115:0/2552909414 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fed08011990 con 0x7fed0c0feda0 2023-07-14T18:10:13.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.528+0000 7fed037fe700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecf80a11a0 0x7fecf80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:13.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.528+0000 7fed1181c700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecf80a11a0 0x7fecf80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:13.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.528+0000 7fed037fe700 1 -- 172.21.15.115:0/2552909414 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fed080cc3d0 con 0x7fed0c0feda0 2023-07-14T18:10:13.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.528+0000 7fed1181c700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecf80a11a0 0x7fecf80a3650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00c010 tx=0x7fecfc00b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:13.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.531+0000 7fed037fe700 1 -- 172.21.15.115:0/2552909414 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fed08095fc0 con 0x7fed0c0feda0 2023-07-14T18:10:13.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:13 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] up:boot 2023-07-14T18:10:13.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:13 smithi115 ceph-mon[127451]: fsmap 12 up:standby 2023-07-14T18:10:13.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:13 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:10:13.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:13 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] up:boot 2023-07-14T18:10:13.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:13 smithi187 ceph-mon[129317]: fsmap 12 up:standby 2023-07-14T18:10:13.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:13 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:10:13.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.775+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed0c105b10 con 0x7fed0c0feda0 2023-07-14T18:10:13.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.776+0000 7fed037fe700 1 -- 172.21.15.115:0/2552909414 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9619 (secure 0 0 0) 0x7fed080995e0 con 0x7fed0c0feda0 2023-07-14T18:10:13.815 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:13.816 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-14T18:10:13.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecf80a11a0 msgr2=0x7fecf80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:13.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecf80a11a0 0x7fecf80a3650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00c010 tx=0x7fecfc00b540 comp rx=0 tx=0).stop 2023-07-14T18:10:13.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 msgr2=0x7fed0c0fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:13.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 0x7fed0c0fe860 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fed0800b500 tx=0x7fed0800b8c0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 shutdown_connections 2023-07-14T18:10:13.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecf80a11a0 0x7fecf80a3650 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fed0c100100 0x7fed0c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fed0c0feda0 0x7fed0c0fe860 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 --2- 172.21.15.115:0/2552909414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed0c0ff750 0x7fed0c19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 >> 172.21.15.115:0/2552909414 conn(0x7fed0c0f98a0 msgr2=0x7fed0c0fbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:13.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 shutdown_connections 2023-07-14T18:10:13.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.780+0000 7fed14281700 1 -- 172.21.15.115:0/2552909414 wait complete. 2023-07-14T18:10:13.820 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 13 2023-07-14T18:10:13.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:10:13.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 -- 172.21.15.115:0/1342318629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941006b0 msgr2=0x7f5994102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:13.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 --2- 172.21.15.115:0/1342318629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941006b0 0x7f5994102a90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5990009990 tx=0x7f5990009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 -- 172.21.15.115:0/1342318629 shutdown_connections 2023-07-14T18:10:13.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 --2- 172.21.15.115:0/1342318629 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59941058f0 0x7f5994107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 --2- 172.21.15.115:0/1342318629 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941006b0 0x7f5994102a90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 --2- 172.21.15.115:0/1342318629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5994102fd0 0x7f59941053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 -- 172.21.15.115:0/1342318629 >> 172.21.15.115:0/1342318629 conn(0x7f59940fa150 msgr2=0x7f59940fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:13.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 -- 172.21.15.115:0/1342318629 shutdown_connections 2023-07-14T18:10:13.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 -- 172.21.15.115:0/1342318629 wait complete. 2023-07-14T18:10:13.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 Processor -- start 2023-07-14T18:10:13.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.919+0000 7f599b54f700 1 -- start start 2023-07-14T18:10:13.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f599b54f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59941006b0 0x7f5994100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:13.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f599b54f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5994102fd0 0x7f59940fe660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:13.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f599b54f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941058f0 0x7f59940feba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:13.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f599b54f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59940ff340 con 0x7f59941058f0 2023-07-14T18:10:13.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f599b54f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59940ff4b0 con 0x7f5994102fd0 2023-07-14T18:10:13.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f599b54f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599419f8b0 con 0x7f59941006b0 2023-07-14T18:10:13.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f5999aec700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941058f0 0x7f59940feba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:13.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f59992eb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59941006b0 0x7f5994100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:13.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f59992eb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59941006b0 0x7f5994100020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.115:51584/0 (socket says 172.21.15.115:51584) 2023-07-14T18:10:13.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f5999aec700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941058f0 0x7f59940feba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52456/0 (socket says 172.21.15.115:52456) 2023-07-14T18:10:13.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f59992eb700 1 -- 172.21.15.115:0/4004419243 learned_addr learned my addr 172.21.15.115:0/4004419243 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:13.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f5998aea700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5994102fd0 0x7f59940fe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:13.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f59992eb700 1 -- 172.21.15.115:0/4004419243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5994102fd0 msgr2=0x7f59940fe660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:13.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f59992eb700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5994102fd0 0x7f59940fe660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f59992eb700 1 -- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941058f0 msgr2=0x7f59940feba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:13.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f59992eb700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941058f0 0x7f59940feba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:13.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f59992eb700 1 -- 172.21.15.115:0/4004419243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59900095f0 con 0x7f59941006b0 2023-07-14T18:10:13.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f5999aec700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941058f0 0x7f59940feba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:10:13.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f59992eb700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59941006b0 0x7f5994100020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f59900098e0 tx=0x7f5990004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:13.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f598a7fc700 1 -- 172.21.15.115:0/4004419243 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599001d070 con 0x7f59941006b0 2023-07-14T18:10:13.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.920+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599419fb30 con 0x7f59941006b0 2023-07-14T18:10:13.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.921+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599419ffb0 con 0x7f59941006b0 2023-07-14T18:10:13.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.921+0000 7f598a7fc700 1 -- 172.21.15.115:0/4004419243 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f599000bdb0 con 0x7f59941006b0 2023-07-14T18:10:13.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.921+0000 7f598a7fc700 1 -- 172.21.15.115:0/4004419243 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990022cd0 con 0x7f59941006b0 2023-07-14T18:10:13.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.921+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599410b2e0 con 0x7f59941006b0 2023-07-14T18:10:13.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.923+0000 7f598a7fc700 1 -- 172.21.15.115:0/4004419243 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f5990022470 con 0x7f59941006b0 2023-07-14T18:10:13.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.923+0000 7f598a7fc700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59800a11a0 0x7f59800a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:13.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.923+0000 7f5998aea700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59800a11a0 0x7f59800a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:13.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.923+0000 7f598a7fc700 1 -- 172.21.15.115:0/4004419243 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f59900cd790 con 0x7f59941006b0 2023-07-14T18:10:13.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.924+0000 7f5998aea700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59800a11a0 0x7f59800a3650 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5984009c60 tx=0x7f5984009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:13.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:13.929+0000 7f598a7fc700 1 -- 172.21.15.115:0/4004419243 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5990097410 con 0x7f59941006b0 2023-07-14T18:10:14.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.123+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59941a0560 con 0x7f59941006b0 2023-07-14T18:10:14.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.125+0000 7f598a7fc700 1 -- 172.21.15.115:0/4004419243 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18492 (secure 0 0 0) 0x7f599009aa30 con 0x7f59941006b0 2023-07-14T18:10:14.163 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:14.164 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":79,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:09:37.624654+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:14.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59800a11a0 msgr2=0x7f59800a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59800a11a0 0x7f59800a3650 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5984009c60 tx=0x7f5984009260 comp rx=0 tx=0).stop 2023-07-14T18:10:14.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59941006b0 msgr2=0x7f5994100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59941006b0 0x7f5994100020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f59900098e0 tx=0x7f5990004730 comp rx=0 tx=0).stop 2023-07-14T18:10:14.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 shutdown_connections 2023-07-14T18:10:14.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59800a11a0 0x7f59800a3650 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59941058f0 0x7f59940feba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59941006b0 0x7f5994100020 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 --2- 172.21.15.115:0/4004419243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5994102fd0 0x7f59940fe660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 >> 172.21.15.115:0/4004419243 conn(0x7f59940fa150 msgr2=0x7f5994101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:14.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 shutdown_connections 2023-07-14T18:10:14.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.127+0000 7f599b54f700 1 -- 172.21.15.115:0/4004419243 wait complete. 2023-07-14T18:10:14.177 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T18:10:14.178 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:10:14.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.270+0000 7f0f013f6700 1 -- 172.21.15.115:0/1022054640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc103070 msgr2=0x7f0efc105450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.270+0000 7f0f013f6700 1 --2- 172.21.15.115:0/1022054640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc103070 0x7f0efc105450 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009990 tx=0x7f0eec009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.270+0000 7f0f013f6700 1 -- 172.21.15.115:0/1022054640 shutdown_connections 2023-07-14T18:10:14.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.270+0000 7f0f013f6700 1 --2- 172.21.15.115:0/1022054640 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0efc105990 0x7f0efc107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.270+0000 7f0f013f6700 1 --2- 172.21.15.115:0/1022054640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc100750 0x7f0efc102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.270+0000 7f0f013f6700 1 --2- 172.21.15.115:0/1022054640 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc103070 0x7f0efc105450 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 -- 172.21.15.115:0/1022054640 >> 172.21.15.115:0/1022054640 conn(0x7f0efc0fa150 msgr2=0x7f0efc0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:14.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 -- 172.21.15.115:0/1022054640 shutdown_connections 2023-07-14T18:10:14.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 -- 172.21.15.115:0/1022054640 wait complete. 2023-07-14T18:10:14.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 Processor -- start 2023-07-14T18:10:14.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 -- start start 2023-07-14T18:10:14.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc100750 0x7f0efc1000c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:14.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0efc103070 0x7f0efc0fe700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:14.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc105990 0x7f0efc0fec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:14.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0ff3e0 con 0x7f0efc105990 2023-07-14T18:10:14.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0ff550 con 0x7f0efc100750 2023-07-14T18:10:14.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.271+0000 7f0f013f6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc19f9c0 con 0x7f0efc103070 2023-07-14T18:10:14.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efb7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc105990 0x7f0efc0fec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:14.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efb7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc105990 0x7f0efc0fec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52468/0 (socket says 172.21.15.115:52468) 2023-07-14T18:10:14.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efb7fe700 1 -- 172.21.15.115:0/3952412997 learned_addr learned my addr 172.21.15.115:0/3952412997 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:14.313 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efaffd700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc100750 0x7f0efc1000c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:14.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efa7fc700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0efc103070 0x7f0efc0fe700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:14.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efb7fe700 1 -- 172.21.15.115:0/3952412997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0efc103070 msgr2=0x7f0efc0fe700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efb7fe700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0efc103070 0x7f0efc0fe700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efb7fe700 1 -- 172.21.15.115:0/3952412997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc100750 msgr2=0x7f0efc1000c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efb7fe700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc100750 0x7f0efc1000c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efb7fe700 1 -- 172.21.15.115:0/3952412997 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec0095f0 con 0x7f0efc105990 2023-07-14T18:10:14.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efaffd700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc100750 0x7f0efc1000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:10:14.314 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0efb7fe700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc105990 0x7f0efc0fec40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400caf0 tx=0x7f0ef400ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:14.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0ee3fff700 1 -- 172.21.15.115:0/3952412997 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4003a00 con 0x7f0efc105990 2023-07-14T18:10:14.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0ee3fff700 1 -- 172.21.15.115:0/3952412997 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0ef400a7e0 con 0x7f0efc105990 2023-07-14T18:10:14.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0ee3fff700 1 -- 172.21.15.115:0/3952412997 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4004430 con 0x7f0efc105990 2023-07-14T18:10:14.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0efc19fca0 con 0x7f0efc105990 2023-07-14T18:10:14.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.272+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc1a0160 con 0x7f0efc105990 2023-07-14T18:10:14.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.273+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0efc0fbcc0 con 0x7f0efc105990 2023-07-14T18:10:14.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.274+0000 7f0ee3fff700 1 -- 172.21.15.115:0/3952412997 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f0ef400a950 con 0x7f0efc105990 2023-07-14T18:10:14.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.275+0000 7f0ee3fff700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ee40a1250 0x7f0ee40a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:14.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.275+0000 7f0efaffd700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ee40a1250 0x7f0ee40a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:14.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.275+0000 7f0ee3fff700 1 -- 172.21.15.115:0/3952412997 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0ef40cc400 con 0x7f0efc105990 2023-07-14T18:10:14.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.275+0000 7f0efaffd700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ee40a1250 0x7f0ee40a3700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0005950 tx=0x7f0ef000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:14.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.278+0000 7f0ee3fff700 1 -- 172.21.15.115:0/3952412997 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ef4095f50 con 0x7f0efc105990 2023-07-14T18:10:14.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.483+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0efc02ca70 con 0x7f0efc105990 2023-07-14T18:10:14.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.484+0000 7f0ee3fff700 1 -- 172.21.15.115:0/3952412997 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ef4099570 con 0x7f0efc105990 2023-07-14T18:10:14.522 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:10:14.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ee40a1250 msgr2=0x7f0ee40a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ee40a1250 0x7f0ee40a3700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0005950 tx=0x7f0ef000b410 comp rx=0 tx=0).stop 2023-07-14T18:10:14.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc105990 msgr2=0x7f0efc0fec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc105990 0x7f0efc0fec40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400caf0 tx=0x7f0ef400ce00 comp rx=0 tx=0).stop 2023-07-14T18:10:14.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 shutdown_connections 2023-07-14T18:10:14.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0ee40a1250 0x7f0ee40a3700 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0efc105990 0x7f0efc0fec40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc100750 0x7f0efc1000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 --2- 172.21.15.115:0/3952412997 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0efc103070 0x7f0efc0fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.487+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 >> 172.21.15.115:0/3952412997 conn(0x7f0efc0fa150 msgr2=0x7f0efc1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:14.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.488+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 shutdown_connections 2023-07-14T18:10:14.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.488+0000 7f0f013f6700 1 -- 172.21.15.115:0/3952412997 wait complete. 2023-07-14T18:10:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:14 smithi115 ceph-mon[127451]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:14 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2552909414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:14 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:14.542 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:14 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4004419243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:14.543 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-14T18:10:14.556 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:14 smithi150 ceph-mon[128853]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:14.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:14 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2552909414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:14.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:14 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:14.557 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:14 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/4004419243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:14.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.636+0000 7f7a4a303700 1 -- 172.21.15.115:0/1977057598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a44100e20 msgr2=0x7f7a44101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.636+0000 7f7a4a303700 1 --2- 172.21.15.115:0/1977057598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a44100e20 0x7f7a44101270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009990 tx=0x7f7a38009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.636+0000 7f7a4a303700 1 -- 172.21.15.115:0/1977057598 shutdown_connections 2023-07-14T18:10:14.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.636+0000 7f7a4a303700 1 --2- 172.21.15.115:0/1977057598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a441017b0 0x7f7a44108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.636+0000 7f7a4a303700 1 --2- 172.21.15.115:0/1977057598 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a440ffc30 0x7f7a44100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.636+0000 7f7a4a303700 1 --2- 172.21.15.115:0/1977057598 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a44100e20 0x7f7a44101270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.636+0000 7f7a4a303700 1 -- 172.21.15.115:0/1977057598 >> 172.21.15.115:0/1977057598 conn(0x7f7a440fb210 msgr2=0x7f7a440fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:14.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.636+0000 7f7a4a303700 1 -- 172.21.15.115:0/1977057598 shutdown_connections 2023-07-14T18:10:14.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.636+0000 7f7a4a303700 1 -- 172.21.15.115:0/1977057598 wait complete. 2023-07-14T18:10:14.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a4a303700 1 Processor -- start 2023-07-14T18:10:14.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a4a303700 1 -- start start 2023-07-14T18:10:14.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a4a303700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a440ffc30 0x7f7a440743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:14.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a4a303700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a44100e20 0x7f7a44072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:14.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a4a303700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a441017b0 0x7f7a44072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:14.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a4a303700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a44073710 con 0x7f7a44100e20 2023-07-14T18:10:14.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a4a303700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a44073880 con 0x7f7a441017b0 2023-07-14T18:10:14.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a4a303700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4419f820 con 0x7f7a440ffc30 2023-07-14T18:10:14.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a437fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a44100e20 0x7f7a44072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:14.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a437fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a44100e20 0x7f7a44072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52478/0 (socket says 172.21.15.115:52478) 2023-07-14T18:10:14.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a437fe700 1 -- 172.21.15.115:0/2197455463 learned_addr learned my addr 172.21.15.115:0/2197455463 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:14.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a43fff700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a440ffc30 0x7f7a440743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:14.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a488a0700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a441017b0 0x7f7a44072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:14.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a437fe700 1 -- 172.21.15.115:0/2197455463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a440ffc30 msgr2=0x7f7a440743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a437fe700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a440ffc30 0x7f7a440743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a437fe700 1 -- 172.21.15.115:0/2197455463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a441017b0 msgr2=0x7f7a44072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a437fe700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a441017b0 0x7f7a44072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.637+0000 7f7a437fe700 1 -- 172.21.15.115:0/2197455463 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a34009520 con 0x7f7a44100e20 2023-07-14T18:10:14.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.638+0000 7f7a437fe700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a44100e20 0x7f7a44072a30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7a380092e0 tx=0x7f7a38003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:14.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.638+0000 7f7a417fa700 1 -- 172.21.15.115:0/2197455463 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3801d070 con 0x7f7a44100e20 2023-07-14T18:10:14.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.638+0000 7f7a417fa700 1 -- 172.21.15.115:0/2197455463 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7a3800be50 con 0x7f7a44100e20 2023-07-14T18:10:14.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.638+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a380095f0 con 0x7f7a44100e20 2023-07-14T18:10:14.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.638+0000 7f7a417fa700 1 -- 172.21.15.115:0/2197455463 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38005270 con 0x7f7a44100e20 2023-07-14T18:10:14.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.638+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4419fd30 con 0x7f7a44100e20 2023-07-14T18:10:14.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.639+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4410b250 con 0x7f7a44100e20 2023-07-14T18:10:14.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.640+0000 7f7a417fa700 1 -- 172.21.15.115:0/2197455463 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7a38004800 con 0x7f7a44100e20 2023-07-14T18:10:14.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.640+0000 7f7a417fa700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a2c0a1220 0x7f7a2c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:14.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.641+0000 7f7a43fff700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a2c0a1220 0x7f7a2c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:14.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.641+0000 7f7a417fa700 1 -- 172.21.15.115:0/2197455463 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7a380cd6c0 con 0x7f7a44100e20 2023-07-14T18:10:14.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:14 smithi187 ceph-mon[129317]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:14.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:14 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2552909414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:14.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:14 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:14.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:14 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/4004419243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:14.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.641+0000 7f7a43fff700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a2c0a1220 0x7f7a2c0a36d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a44100c80 tx=0x7f7a34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:14.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.645+0000 7f7a417fa700 1 -- 172.21.15.115:0/2197455463 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7a3805a780 con 0x7f7a44100e20 2023-07-14T18:10:14.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.862+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7a441a03d0 con 0x7f7a44100e20 2023-07-14T18:10:14.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.863+0000 7f7a417fa700 1 -- 172.21.15.115:0/2197455463 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7a380972a0 con 0x7f7a44100e20 2023-07-14T18:10:14.901 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:14.902 INFO:teuthology.orchestra.run.smithi115.stdout:[ 2023-07-14T18:10:14.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2023-07-14T18:10:14.902 INFO:teuthology.orchestra.run.smithi115.stdout: "poolnum": 1, 2023-07-14T18:10:14.902 INFO:teuthology.orchestra.run.smithi115.stdout: "poolname": ".mgr" 2023-07-14T18:10:14.902 INFO:teuthology.orchestra.run.smithi115.stdout: } 2023-07-14T18:10:14.902 INFO:teuthology.orchestra.run.smithi115.stdout:] 2023-07-14T18:10:14.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.864+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a2c0a1220 msgr2=0x7f7a2c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.864+0000 7f7a4a303700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a2c0a1220 0x7f7a2c0a36d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a44100c80 tx=0x7f7a34009260 comp rx=0 tx=0).stop 2023-07-14T18:10:14.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.864+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a44100e20 msgr2=0x7f7a44072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:14.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.865+0000 7f7a4a303700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a44100e20 0x7f7a44072a30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7a380092e0 tx=0x7f7a38003920 comp rx=0 tx=0).stop 2023-07-14T18:10:14.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.865+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 shutdown_connections 2023-07-14T18:10:14.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.865+0000 7f7a4a303700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a2c0a1220 0x7f7a2c0a36d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.865+0000 7f7a4a303700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a441017b0 0x7f7a44072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.865+0000 7f7a4a303700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a440ffc30 0x7f7a440743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.865+0000 7f7a4a303700 1 --2- 172.21.15.115:0/2197455463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a44100e20 0x7f7a44072a30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:14.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.865+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 >> 172.21.15.115:0/2197455463 conn(0x7f7a440fb210 msgr2=0x7f7a440fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:14.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.865+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 shutdown_connections 2023-07-14T18:10:14.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:14.865+0000 7f7a4a303700 1 -- 172.21.15.115:0/2197455463 wait complete. 2023-07-14T18:10:14.915 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-14T18:10:14.916 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-14T18:10:15.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 -- 172.21.15.115:0/2917970509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 msgr2=0x7f827c100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:15.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 --2- 172.21.15.115:0/2917970509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 0x7f827c100020 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f826c009990 tx=0x7f826c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:15.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 -- 172.21.15.115:0/2917970509 shutdown_connections 2023-07-14T18:10:15.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 --2- 172.21.15.115:0/2917970509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f827c101790 0x7f827c108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:15.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 --2- 172.21.15.115:0/2917970509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 0x7f827c100020 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:15.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 --2- 172.21.15.115:0/2917970509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f827c100e00 0x7f827c101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:15.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 -- 172.21.15.115:0/2917970509 >> 172.21.15.115:0/2917970509 conn(0x7f827c0fb250 msgr2=0x7f827c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:15.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 -- 172.21.15.115:0/2917970509 shutdown_connections 2023-07-14T18:10:15.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 -- 172.21.15.115:0/2917970509 wait complete. 2023-07-14T18:10:15.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.024+0000 7f8281c79700 1 Processor -- start 2023-07-14T18:10:15.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f8281c79700 1 -- start start 2023-07-14T18:10:15.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f8281c79700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 0x7f827c1958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:15.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f8281c79700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f827c100e00 0x7f827c195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:15.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f8281c79700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f827c101790 0x7f827c19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:15.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f8281c79700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c19d460 con 0x7f827c0ffc10 2023-07-14T18:10:15.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f8281c79700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c19d5a0 con 0x7f827c101790 2023-07-14T18:10:15.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f8281c79700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c19d730 con 0x7f827c100e00 2023-07-14T18:10:15.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 0x7f827c1958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:15.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 0x7f827c1958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52490/0 (socket says 172.21.15.115:52490) 2023-07-14T18:10:15.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827b7fe700 1 -- 172.21.15.115:0/1258702055 learned_addr learned my addr 172.21.15.115:0/1258702055 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:15.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827affd700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f827c100e00 0x7f827c195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:15.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827bfff700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f827c101790 0x7f827c19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:15.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827b7fe700 1 -- 172.21.15.115:0/1258702055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f827c100e00 msgr2=0x7f827c195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:15.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827b7fe700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f827c100e00 0x7f827c195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:15.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827b7fe700 1 -- 172.21.15.115:0/1258702055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f827c101790 msgr2=0x7f827c19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:15.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827b7fe700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f827c101790 0x7f827c19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:15.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827b7fe700 1 -- 172.21.15.115:0/1258702055 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0095f0 con 0x7f827c0ffc10 2023-07-14T18:10:15.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f827b7fe700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 0x7f827c1958e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f826c004730 tx=0x7f826c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:15.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f8278ff9700 1 -- 172.21.15.115:0/1258702055 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c01d070 con 0x7f827c0ffc10 2023-07-14T18:10:15.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.025+0000 7f8278ff9700 1 -- 172.21.15.115:0/1258702055 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f826c004c60 con 0x7f827c0ffc10 2023-07-14T18:10:15.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.026+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c19d9b0 con 0x7f827c0ffc10 2023-07-14T18:10:15.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.026+0000 7f8278ff9700 1 -- 172.21.15.115:0/1258702055 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c003af0 con 0x7f827c0ffc10 2023-07-14T18:10:15.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.026+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c19ddd0 con 0x7f827c0ffc10 2023-07-14T18:10:15.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.027+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827c10b2e0 con 0x7f827c0ffc10 2023-07-14T18:10:15.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.027+0000 7f8278ff9700 1 -- 172.21.15.115:0/1258702055 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f826c0047d0 con 0x7f827c0ffc10 2023-07-14T18:10:15.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.028+0000 7f8278ff9700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82640a1150 0x7f82640a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:15.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.028+0000 7f8278ff9700 1 -- 172.21.15.115:0/1258702055 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f826c0cd350 con 0x7f827c0ffc10 2023-07-14T18:10:15.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.028+0000 7f827affd700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82640a1150 0x7f82640a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:15.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.029+0000 7f827affd700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82640a1150 0x7f82640a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8270005950 tx=0x7f827000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:15.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.032+0000 7f8278ff9700 1 -- 172.21.15.115:0/1258702055 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f826c096fd0 con 0x7f827c0ffc10 2023-07-14T18:10:15.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:15.236+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f827c106be0 con 0x7f827c0ffc10 2023-07-14T18:10:15.608 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3952412997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:10:15.609 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.609 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.609 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.609 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.609 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.609 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2197455463' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T18:10:15.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:15.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:10:15.610 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:15 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3952412997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:10:15.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2197455463' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T18:10:15.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:15.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:10:15.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:15 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3952412997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:10:15.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:15.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2197455463' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T18:10:15.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:15.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:10:15.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:15 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:16.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.190+0000 7f8278ff9700 1 -- 172.21.15.115:0/1258702055 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7f826c09a5f0 con 0x7f827c0ffc10 2023-07-14T18:10:16.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82640a1150 msgr2=0x7f82640a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:16.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82640a1150 0x7f82640a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8270005950 tx=0x7f827000a300 comp rx=0 tx=0).stop 2023-07-14T18:10:16.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 msgr2=0x7f827c1958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:16.232 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 0x7f827c1958e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f826c004730 tx=0x7f826c009580 comp rx=0 tx=0).stop 2023-07-14T18:10:16.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 shutdown_connections 2023-07-14T18:10:16.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82640a1150 0x7f82640a3600 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f827c101790 0x7f827c19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f827c0ffc10 0x7f827c1958e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 --2- 172.21.15.115:0/1258702055 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f827c100e00 0x7f827c195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.233 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.194+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 >> 172.21.15.115:0/1258702055 conn(0x7f827c0fb250 msgr2=0x7f827c0fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:16.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.195+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 shutdown_connections 2023-07-14T18:10:16.234 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.195+0000 7f8281c79700 1 -- 172.21.15.115:0/1258702055 wait complete. 2023-07-14T18:10:16.234 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_metadata' created 2023-07-14T18:10:16.245 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-07-14T18:10:16.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.339+0000 7f5d615a1700 1 -- 172.21.15.115:0/1535763955 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c101260 msgr2=0x7f5d5c101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:16.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.339+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1535763955 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c101260 0x7f5d5c101670 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009990 tx=0x7f5d4c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.339+0000 7f5d615a1700 1 -- 172.21.15.115:0/1535763955 shutdown_connections 2023-07-14T18:10:16.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.339+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1535763955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d5c0fe660 0x7f5d5c108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.339+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1535763955 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c101260 0x7f5d5c101670 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.339+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1535763955 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d5c0fdcd0 0x7f5d5c0fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.339+0000 7f5d615a1700 1 -- 172.21.15.115:0/1535763955 >> 172.21.15.115:0/1535763955 conn(0x7f5d5c0f98e0 msgr2=0x7f5d5c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:16.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.339+0000 7f5d615a1700 1 -- 172.21.15.115:0/1535763955 shutdown_connections 2023-07-14T18:10:16.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.339+0000 7f5d615a1700 1 -- 172.21.15.115:0/1535763955 wait complete. 2023-07-14T18:10:16.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d615a1700 1 Processor -- start 2023-07-14T18:10:16.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d615a1700 1 -- start start 2023-07-14T18:10:16.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d615a1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d5c0fdcd0 0x7f5d5c1958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:16.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d615a1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c0fe660 0x7f5d5c195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:16.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d615a1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d5c101260 0x7f5d5c19ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:16.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d615a1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c19d420 con 0x7f5d5c0fe660 2023-07-14T18:10:16.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d615a1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c19d560 con 0x7f5d5c101260 2023-07-14T18:10:16.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d615a1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c19d6f0 con 0x7f5d5c0fdcd0 2023-07-14T18:10:16.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d5a7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c0fe660 0x7f5d5c195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:16.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d5a7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c0fe660 0x7f5d5c195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52506/0 (socket says 172.21.15.115:52506) 2023-07-14T18:10:16.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d5a7fc700 1 -- 172.21.15.115:0/1733086697 learned_addr learned my addr 172.21.15.115:0/1733086697 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:16.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d5b7fe700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d5c101260 0x7f5d5c19ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:16.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.340+0000 7f5d5affd700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d5c0fdcd0 0x7f5d5c1958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:16.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d5a7fc700 1 -- 172.21.15.115:0/1733086697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d5c0fdcd0 msgr2=0x7f5d5c1958a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:16.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d5a7fc700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d5c0fdcd0 0x7f5d5c1958a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d5a7fc700 1 -- 172.21.15.115:0/1733086697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d5c101260 msgr2=0x7f5d5c19ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:16.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d5a7fc700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d5c101260 0x7f5d5c19ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:16.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d5a7fc700 1 -- 172.21.15.115:0/1733086697 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c0095f0 con 0x7f5d5c0fe660 2023-07-14T18:10:16.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d5b7fe700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d5c101260 0x7f5d5c19ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:10:16.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d5a7fc700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c0fe660 0x7f5d5c195de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000d950 tx=0x7f5d5000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:16.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d43fff700 1 -- 172.21.15.115:0/1733086697 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5000f890 con 0x7f5d5c0fe660 2023-07-14T18:10:16.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d43fff700 1 -- 172.21.15.115:0/1733086697 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5d5000fe80 con 0x7f5d5c0fe660 2023-07-14T18:10:16.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c19d9d0 con 0x7f5d5c0fe660 2023-07-14T18:10:16.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d43fff700 1 -- 172.21.15.115:0/1733086697 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5000e3b0 con 0x7f5d5c0fe660 2023-07-14T18:10:16.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.341+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c19dee0 con 0x7f5d5c0fe660 2023-07-14T18:10:16.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.342+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c10b2c0 con 0x7f5d5c0fe660 2023-07-14T18:10:16.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.343+0000 7f5d43fff700 1 -- 172.21.15.115:0/1733086697 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f5d50010460 con 0x7f5d5c0fe660 2023-07-14T18:10:16.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.344+0000 7f5d43fff700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5d440a1150 0x7f5d440a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:16.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.344+0000 7f5d5affd700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5d440a1150 0x7f5d440a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:16.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.344+0000 7f5d43fff700 1 -- 172.21.15.115:0/1733086697 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f5d500cc360 con 0x7f5d5c0fe660 2023-07-14T18:10:16.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.344+0000 7f5d5affd700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5d440a1150 0x7f5d440a3600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c005910 tx=0x7f5d4c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:16.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.347+0000 7f5d43fff700 1 -- 172.21.15.115:0/1733086697 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5d50095de0 con 0x7f5d5c0fe660 2023-07-14T18:10:16.587 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:16.549+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f5d5c106d50 con 0x7f5d5c0fe660 2023-07-14T18:10:16.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:16 smithi115 ceph-mon[127451]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:16.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:16 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1258702055' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T18:10:16.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:16 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:10:16.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:16 smithi150 ceph-mon[128853]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:16.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:16 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1258702055' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T18:10:16.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:16 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:10:16.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:16 smithi187 ceph-mon[129317]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:16.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:16 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1258702055' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T18:10:16.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:16 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:10:17.235 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.197+0000 7f5d43fff700 1 -- 172.21.15.115:0/1733086697 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v81) v1 ==== 171+0+0 (secure 0 0 0) 0x7f5d50099400 con 0x7f5d5c0fe660 2023-07-14T18:10:17.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.203+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5d440a1150 msgr2=0x7f5d440a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:17.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.203+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5d440a1150 0x7f5d440a3600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c005910 tx=0x7f5d4c005960 comp rx=0 tx=0).stop 2023-07-14T18:10:17.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.203+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c0fe660 msgr2=0x7f5d5c195de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:17.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.203+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c0fe660 0x7f5d5c195de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000d950 tx=0x7f5d5000dc60 comp rx=0 tx=0).stop 2023-07-14T18:10:17.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.207+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 shutdown_connections 2023-07-14T18:10:17.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.207+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5d440a1150 0x7f5d440a3600 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.207+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d5c101260 0x7f5d5c19ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.207+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d5c0fdcd0 0x7f5d5c1958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.207+0000 7f5d615a1700 1 --2- 172.21.15.115:0/1733086697 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5d5c0fe660 0x7f5d5c195de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.207+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 >> 172.21.15.115:0/1733086697 conn(0x7f5d5c0f98e0 msgr2=0x7f5d5c0fbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:17.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.212+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 shutdown_connections 2023-07-14T18:10:17.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.212+0000 7f5d615a1700 1 -- 172.21.15.115:0/1733086697 wait complete. 2023-07-14T18:10:17.254 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_data' created 2023-07-14T18:10:17.265 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-14T18:10:17.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.355+0000 7fe522e34700 1 -- 172.21.15.115:0/1459678785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0ffdc0 msgr2=0x7fe51c100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:17.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.355+0000 7fe522e34700 1 --2- 172.21.15.115:0/1459678785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0ffdc0 0x7fe51c100230 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe508009990 tx=0x7fe508009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 -- 172.21.15.115:0/1459678785 shutdown_connections 2023-07-14T18:10:17.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 --2- 172.21.15.115:0/1459678785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe51c100770 0x7fe51c107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 --2- 172.21.15.115:0/1459678785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe51c0feb30 0x7fe51c0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 --2- 172.21.15.115:0/1459678785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0ffdc0 0x7fe51c100230 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 -- 172.21.15.115:0/1459678785 >> 172.21.15.115:0/1459678785 conn(0x7fe51c0fa150 msgr2=0x7fe51c0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:17.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 -- 172.21.15.115:0/1459678785 shutdown_connections 2023-07-14T18:10:17.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 -- 172.21.15.115:0/1459678785 wait complete. 2023-07-14T18:10:17.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 Processor -- start 2023-07-14T18:10:17.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 -- start start 2023-07-14T18:10:17.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0feb30 0x7fe51c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:17.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe51c0ffdc0 0x7fe51c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe51c100770 0x7fe51c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c19f580 con 0x7fe51c0feb30 2023-07-14T18:10:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c19f6c0 con 0x7fe51c0ffdc0 2023-07-14T18:10:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe522e34700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51c19f850 con 0x7fe51c100770 2023-07-14T18:10:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe520bd0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0feb30 0x7fe51c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe520bd0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0feb30 0x7fe51c197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52522/0 (socket says 172.21.15.115:52522) 2023-07-14T18:10:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe520bd0700 1 -- 172.21.15.115:0/659491812 learned_addr learned my addr 172.21.15.115:0/659491812 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe5213d1700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe51c100770 0x7fe51c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:17.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.356+0000 7fe51bfff700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe51c0ffdc0 0x7fe51c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:17.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe520bd0700 1 -- 172.21.15.115:0/659491812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe51c100770 msgr2=0x7fe51c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:17.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe520bd0700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe51c100770 0x7fe51c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe520bd0700 1 -- 172.21.15.115:0/659491812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe51c0ffdc0 msgr2=0x7fe51c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:17.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe520bd0700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe51c0ffdc0 0x7fe51c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:17.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe520bd0700 1 -- 172.21.15.115:0/659491812 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5080095f0 con 0x7fe51c0feb30 2023-07-14T18:10:17.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe520bd0700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0feb30 0x7fe51c197a00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe51000e9a0 tx=0x7fe51000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:17.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe519ffb700 1 -- 172.21.15.115:0/659491812 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510009790 con 0x7fe51c0feb30 2023-07-14T18:10:17.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe519ffb700 1 -- 172.21.15.115:0/659491812 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe510010430 con 0x7fe51c0feb30 2023-07-14T18:10:17.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51c19fb30 con 0x7fe51c0feb30 2023-07-14T18:10:17.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe519ffb700 1 -- 172.21.15.115:0/659491812 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51000cea0 con 0x7fe51c0feb30 2023-07-14T18:10:17.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.357+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51c1a0040 con 0x7fe51c0feb30 2023-07-14T18:10:17.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.358+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51c10a1b0 con 0x7fe51c0feb30 2023-07-14T18:10:17.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.360+0000 7fe519ffb700 1 -- 172.21.15.115:0/659491812 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fe5100105a0 con 0x7fe51c0feb30 2023-07-14T18:10:17.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.360+0000 7fe519ffb700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe50c0a1150 0x7fe50c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:17.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.360+0000 7fe519ffb700 1 -- 172.21.15.115:0/659491812 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fe510014070 con 0x7fe51c0feb30 2023-07-14T18:10:17.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.360+0000 7fe51bfff700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe50c0a1150 0x7fe50c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:17.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.361+0000 7fe51bfff700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe50c0a1150 0x7fe50c0a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe5080059d0 tx=0x7fe508005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:17.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.364+0000 7fe519ffb700 1 -- 172.21.15.115:0/659491812 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe5100963c0 con 0x7fe51c0feb30 2023-07-14T18:10:17.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:17 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1258702055' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T18:10:17.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:17 smithi115 ceph-mon[127451]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T18:10:17.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:17 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1733086697' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T18:10:17.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:17 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1258702055' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T18:10:17.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:17 smithi150 ceph-mon[128853]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T18:10:17.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:17 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1733086697' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T18:10:17.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:17.632+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe51c1a0740 con 0x7fe51c0feb30 2023-07-14T18:10:17.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:17 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1258702055' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T18:10:17.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:17 smithi187 ceph-mon[129317]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T18:10:17.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:17 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1733086697' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T18:10:18.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.411+0000 7fe519ffb700 1 -- 172.21.15.115:0/659491812 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe5100260c0 con 0x7fe51c0feb30 2023-07-14T18:10:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe50c0a1150 msgr2=0x7fe50c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe50c0a1150 0x7fe50c0a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe5080059d0 tx=0x7fe508005960 comp rx=0 tx=0).stop 2023-07-14T18:10:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0feb30 msgr2=0x7fe51c197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0feb30 0x7fe51c197a00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe51000e9a0 tx=0x7fe51000ecb0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 shutdown_connections 2023-07-14T18:10:18.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe50c0a1150 0x7fe50c0a3600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe51c100770 0x7fe51c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe51c0feb30 0x7fe51c197a00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 --2- 172.21.15.115:0/659491812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe51c0ffdc0 0x7fe51c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 >> 172.21.15.115:0/659491812 conn(0x7fe51c0fa150 msgr2=0x7fe51c0689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 shutdown_connections 2023-07-14T18:10:18.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.413+0000 7fe522e34700 1 -- 172.21.15.115:0/659491812 wait complete. 2023-07-14T18:10:18.454 INFO:teuthology.orchestra.run.smithi115.stderr:new fs with metadata pool 2 and data pool 3 2023-07-14T18:10:18.466 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-07-14T18:10:18.467 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-07-14T18:10:18.490 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:18 smithi115 ceph-mon[127451]: pgmap v244: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:18.490 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:18 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1733086697' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T18:10:18.490 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:18 smithi115 ceph-mon[127451]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T18:10:18.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:18 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/659491812' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T18:10:18.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:18 smithi115 ceph-mon[127451]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T18:10:18.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:18 smithi115 ceph-mon[127451]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T18:10:18.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:18 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:18.491 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:18 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:18.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 -- 172.21.15.115:0/3017816255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c1006b0 msgr2=0x7faa9c106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:18.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 --2- 172.21.15.115:0/3017816255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c1006b0 0x7faa9c106f70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faa9400b0a0 tx=0x7faa9400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 -- 172.21.15.115:0/3017816255 shutdown_connections 2023-07-14T18:10:18.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 --2- 172.21.15.115:0/3017816255 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c1006b0 0x7faa9c106f70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 --2- 172.21.15.115:0/3017816255 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa9c0feb30 0x7faa9c0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 --2- 172.21.15.115:0/3017816255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c0ffd20 0x7faa9c100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 -- 172.21.15.115:0/3017816255 >> 172.21.15.115:0/3017816255 conn(0x7faa9c0fa150 msgr2=0x7faa9c0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:18.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 -- 172.21.15.115:0/3017816255 shutdown_connections 2023-07-14T18:10:18.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 -- 172.21.15.115:0/3017816255 wait complete. 2023-07-14T18:10:18.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.556+0000 7faaa1318700 1 Processor -- start 2023-07-14T18:10:18.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faaa1318700 1 -- start start 2023-07-14T18:10:18.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faaa1318700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c0feb30 0x7faa9c1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:18.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faaa1318700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c0ffd20 0x7faa9c195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:18.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faaa1318700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa9c1006b0 0x7faa9c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:18.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faaa1318700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c19d520 con 0x7faa9c0feb30 2023-07-14T18:10:18.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faaa1318700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c19d660 con 0x7faa9c0ffd20 2023-07-14T18:10:18.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faaa1318700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c19d7f0 con 0x7faa9c1006b0 2023-07-14T18:10:18.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c0feb30 0x7faa9c1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:18.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c0feb30 0x7faa9c1959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52538/0 (socket says 172.21.15.115:52538) 2023-07-14T18:10:18.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9affd700 1 -- 172.21.15.115:0/228082188 learned_addr learned my addr 172.21.15.115:0/228082188 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:18.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9a7fc700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c0ffd20 0x7faa9c195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:18.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9b7fe700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa9c1006b0 0x7faa9c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:18.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9affd700 1 -- 172.21.15.115:0/228082188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa9c1006b0 msgr2=0x7faa9c19cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:18.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9affd700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa9c1006b0 0x7faa9c19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9affd700 1 -- 172.21.15.115:0/228082188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c0ffd20 msgr2=0x7faa9c195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:18.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9affd700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c0ffd20 0x7faa9c195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:18.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9affd700 1 -- 172.21.15.115:0/228082188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa94009d00 con 0x7faa9c0feb30 2023-07-14T18:10:18.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa9affd700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c0feb30 0x7faa9c1959a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faa9000e970 tx=0x7faa9000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:18.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa83fff700 1 -- 172.21.15.115:0/228082188 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90009790 con 0x7faa9c0feb30 2023-07-14T18:10:18.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa83fff700 1 -- 172.21.15.115:0/228082188 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faa9000eec0 con 0x7faa9c0feb30 2023-07-14T18:10:18.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c19dad0 con 0x7faa9c0feb30 2023-07-14T18:10:18.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faa83fff700 1 -- 172.21.15.115:0/228082188 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa90003c10 con 0x7faa9c0feb30 2023-07-14T18:10:18.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.557+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c19dfe0 con 0x7faa9c0feb30 2023-07-14T18:10:18.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.558+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c10a1c0 con 0x7faa9c0feb30 2023-07-14T18:10:18.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.559+0000 7faa83fff700 1 -- 172.21.15.115:0/228082188 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7faa90003d90 con 0x7faa9c0feb30 2023-07-14T18:10:18.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.560+0000 7faa83fff700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faa840a1150 0x7faa840a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:18.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.560+0000 7faa9a7fc700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faa840a1150 0x7faa840a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:18.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.560+0000 7faa83fff700 1 -- 172.21.15.115:0/228082188 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7faa90014070 con 0x7faa9c0feb30 2023-07-14T18:10:18.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.560+0000 7faa9a7fc700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faa840a1150 0x7faa840a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009c60 tx=0x7faa8c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:18.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.563+0000 7faa83fff700 1 -- 172.21.15.115:0/228082188 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faa90096440 con 0x7faa9c0feb30 2023-07-14T18:10:18.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:18 smithi150 ceph-mon[128853]: pgmap v244: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:18.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:18 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1733086697' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T18:10:18.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:18 smithi150 ceph-mon[128853]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T18:10:18.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:18 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/659491812' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T18:10:18.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:18 smithi150 ceph-mon[128853]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T18:10:18.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:18 smithi150 ceph-mon[128853]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T18:10:18.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:18 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:18.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:18 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:18.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:18 smithi187 ceph-mon[129317]: pgmap v244: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:10:18.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:18 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1733086697' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T18:10:18.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:18 smithi187 ceph-mon[129317]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T18:10:18.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:18 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/659491812' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T18:10:18.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:18 smithi187 ceph-mon[129317]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T18:10:18.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:18 smithi187 ceph-mon[129317]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T18:10:18.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:18 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:18.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:18 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:18.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:18.765+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7faa9c19e5f0 con 0x7faa9c0feb30 2023-07-14T18:10:19.135 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.097+0000 7faa83fff700 1 -- 172.21.15.115:0/228082188 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v83) v1 ==== 157+0+0 (secure 0 0 0) 0x7faa90099a60 con 0x7faa9c0feb30 2023-07-14T18:10:19.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.099+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faa840a1150 msgr2=0x7faa840a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:19.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.099+0000 7faaa1318700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faa840a1150 0x7faa840a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009c60 tx=0x7faa8c009260 comp rx=0 tx=0).stop 2023-07-14T18:10:19.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.099+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c0feb30 msgr2=0x7faa9c1959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:19.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.099+0000 7faaa1318700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c0feb30 0x7faa9c1959a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faa9000e970 tx=0x7faa9000ec80 comp rx=0 tx=0).stop 2023-07-14T18:10:19.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.099+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 shutdown_connections 2023-07-14T18:10:19.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.100+0000 7faaa1318700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faa840a1150 0x7faa840a3600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.100+0000 7faaa1318700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faa9c1006b0 0x7faa9c19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.100+0000 7faaa1318700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faa9c0feb30 0x7faa9c1959a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.100+0000 7faaa1318700 1 --2- 172.21.15.115:0/228082188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa9c0ffd20 0x7faa9c195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.100+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 >> 172.21.15.115:0/228082188 conn(0x7faa9c0fa150 msgr2=0x7faa9c068a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:19.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.100+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 shutdown_connections 2023-07-14T18:10:19.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.100+0000 7faaa1318700 1 -- 172.21.15.115:0/228082188 wait complete. 2023-07-14T18:10:19.150 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-07-14T18:10:19.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.284+0000 7f37bcb6f700 1 -- 172.21.15.115:0/4188806752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 msgr2=0x7f37b8100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:19.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.284+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/4188806752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 0x7f37b8100170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009990 tx=0x7f37a8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.284+0000 7f37bcb6f700 1 -- 172.21.15.115:0/4188806752 shutdown_connections 2023-07-14T18:10:19.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.284+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/4188806752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b81006b0 0x7f37b8106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.284+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/4188806752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b80feb30 0x7f37b80fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.284+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/4188806752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 0x7f37b8100170 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.284+0000 7f37bcb6f700 1 -- 172.21.15.115:0/4188806752 >> 172.21.15.115:0/4188806752 conn(0x7f37b80fa110 msgr2=0x7f37b80fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:19.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.284+0000 7f37bcb6f700 1 -- 172.21.15.115:0/4188806752 shutdown_connections 2023-07-14T18:10:19.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.284+0000 7f37bcb6f700 1 -- 172.21.15.115:0/4188806752 wait complete. 2023-07-14T18:10:19.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37bcb6f700 1 Processor -- start 2023-07-14T18:10:19.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37bcb6f700 1 -- start start 2023-07-14T18:10:19.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37bcb6f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b80feb30 0x7f37b8195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:19.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37bcb6f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 0x7f37b8195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:19.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37bcb6f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b81006b0 0x7f37b819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:19.326 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37bcb6f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b819d3c0 con 0x7f37b80ffd20 2023-07-14T18:10:19.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37bcb6f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b819d500 con 0x7f37b81006b0 2023-07-14T18:10:19.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37bcb6f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b819d690 con 0x7f37b80feb30 2023-07-14T18:10:19.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b5d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 0x7f37b8195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:19.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b5d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 0x7f37b8195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52554/0 (socket says 172.21.15.115:52554) 2023-07-14T18:10:19.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b5d9b700 1 -- 172.21.15.115:0/2937769424 learned_addr learned my addr 172.21.15.115:0/2937769424 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:19.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b6d9d700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b81006b0 0x7f37b819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:19.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b659c700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b80feb30 0x7f37b8195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b5d9b700 1 -- 172.21.15.115:0/2937769424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b80feb30 msgr2=0x7f37b8195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b5d9b700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b80feb30 0x7f37b8195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b5d9b700 1 -- 172.21.15.115:0/2937769424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b81006b0 msgr2=0x7f37b819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b5d9b700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b81006b0 0x7f37b819cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b5d9b700 1 -- 172.21.15.115:0/2937769424 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a80095f0 con 0x7f37b80ffd20 2023-07-14T18:10:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b6d9d700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b81006b0 0x7f37b819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:10:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37b5d9b700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 0x7f37b8195d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f37a8005f50 tx=0x7f37a80047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.285+0000 7f37a77fe700 1 -- 172.21.15.115:0/2937769424 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a801d070 con 0x7f37b80ffd20 2023-07-14T18:10:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.286+0000 7f37a77fe700 1 -- 172.21.15.115:0/2937769424 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f37a8004810 con 0x7f37b80ffd20 2023-07-14T18:10:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.286+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b819d910 con 0x7f37b80ffd20 2023-07-14T18:10:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.286+0000 7f37a77fe700 1 -- 172.21.15.115:0/2937769424 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8005340 con 0x7f37b80ffd20 2023-07-14T18:10:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.286+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b819dd30 con 0x7f37b80ffd20 2023-07-14T18:10:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.287+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b810a1b0 con 0x7f37b80ffd20 2023-07-14T18:10:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.288+0000 7f37a77fe700 1 -- 172.21.15.115:0/2937769424 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f37a8004980 con 0x7f37b80ffd20 2023-07-14T18:10:19.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.288+0000 7f37a77fe700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f37a00a11f0 0x7f37a00a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:19.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.288+0000 7f37b659c700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f37a00a11f0 0x7f37a00a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.288+0000 7f37a77fe700 1 -- 172.21.15.115:0/2937769424 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f37a80cdf00 con 0x7f37b80ffd20 2023-07-14T18:10:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.289+0000 7f37b659c700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f37a00a11f0 0x7f37a00a36a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f37ac005fd0 tx=0x7f37ac005dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.292+0000 7f37a77fe700 1 -- 172.21.15.115:0/2937769424 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f37a8097700 con 0x7f37b80ffd20 2023-07-14T18:10:19.521 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:19.483+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f37b810a370 con 0x7f37b80ffd20 2023-07-14T18:10:19.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:19.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:10:19.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/659491812' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T18:10:19.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T18:10:19.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T18:10:19.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/228082188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T18:10:19.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/228082188' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T18:10:19.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:19 smithi115 ceph-mon[127451]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T18:10:19.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:19.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:10:19.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/659491812' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T18:10:19.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T18:10:19.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T18:10:19.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/228082188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T18:10:19.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/228082188' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T18:10:19.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:19 smithi150 ceph-mon[128853]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T18:10:19.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:10:19.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:10:19.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:10:19.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/659491812' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T18:10:19.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T18:10:19.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T18:10:19.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/228082188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T18:10:19.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/228082188' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T18:10:19.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:19 smithi187 ceph-mon[129317]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T18:10:20.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:20 smithi115 ceph-mon[127451]: pgmap v247: 97 pgs: 28 active+clean, 28 creating+peering, 41 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T18:10:20.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:20 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] up:active 2023-07-14T18:10:20.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:20 smithi115 ceph-mon[127451]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:20.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:20 smithi115 ceph-mon[127451]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:20.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:20 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2937769424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T18:10:20.620 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:20 smithi150 ceph-mon[128853]: pgmap v247: 97 pgs: 28 active+clean, 28 creating+peering, 41 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T18:10:20.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:20 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] up:active 2023-07-14T18:10:20.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:20 smithi150 ceph-mon[128853]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:20.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:20 smithi150 ceph-mon[128853]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:20.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:20 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2937769424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T18:10:20.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:20 smithi187 ceph-mon[129317]: pgmap v247: 97 pgs: 28 active+clean, 28 creating+peering, 41 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T18:10:20.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:20 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] up:active 2023-07-14T18:10:20.683 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:20 smithi187 ceph-mon[129317]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:20.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:20 smithi187 ceph-mon[129317]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:20.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:20 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2937769424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T18:10:21.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.429+0000 7f37a77fe700 1 -- 172.21.15.115:0/2937769424 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v85) v1 ==== 228+0+0 (secure 0 0 0) 0x7f37a809ad20 con 0x7f37b80ffd20 2023-07-14T18:10:21.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.433+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f37a00a11f0 msgr2=0x7f37a00a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:21.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.433+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f37a00a11f0 0x7f37a00a36a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f37ac005fd0 tx=0x7f37ac005dd0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.433+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 msgr2=0x7f37b8195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:21.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.433+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 0x7f37b8195d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f37a8005f50 tx=0x7f37a80047a0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.434+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 shutdown_connections 2023-07-14T18:10:21.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.434+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f37a00a11f0 0x7f37a00a36a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.434+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b81006b0 0x7f37b819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.434+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f37b80feb30 0x7f37b8195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.434+0000 7f37bcb6f700 1 --2- 172.21.15.115:0/2937769424 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f37b80ffd20 0x7f37b8195d80 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f37a8005f50 tx=0x7f37a80047a0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.434+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 >> 172.21.15.115:0/2937769424 conn(0x7f37b80fa110 msgr2=0x7f37b80687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:21.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.438+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 shutdown_connections 2023-07-14T18:10:21.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.439+0000 7f37bcb6f700 1 -- 172.21.15.115:0/2937769424 wait complete. 2023-07-14T18:10:21.478 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_data_ec' created 2023-07-14T18:10:21.492 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-07-14T18:10:21.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 -- 172.21.15.115:0/163973700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0feb40 msgr2=0x7f7fcc0fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:21.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/163973700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0feb40 0x7f7fcc0fef50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc009990 tx=0x7f7fbc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 -- 172.21.15.115:0/163973700 shutdown_connections 2023-07-14T18:10:21.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/163973700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fcc1006c0 0x7f7fcc106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/163973700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0feb40 0x7f7fcc0fef50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/163973700 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fcc0ffd30 0x7f7fcc100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 -- 172.21.15.115:0/163973700 >> 172.21.15.115:0/163973700 conn(0x7f7fcc0fa160 msgr2=0x7f7fcc0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:21.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 -- 172.21.15.115:0/163973700 shutdown_connections 2023-07-14T18:10:21.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 -- 172.21.15.115:0/163973700 wait complete. 2023-07-14T18:10:21.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.585+0000 7f7fd28cf700 1 Processor -- start 2023-07-14T18:10:21.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd28cf700 1 -- start start 2023-07-14T18:10:21.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd28cf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fcc0feb40 0x7f7fcc195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:21.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd28cf700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0ffd30 0x7f7fcc195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:21.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd28cf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fcc1006c0 0x7f7fcc19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:21.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd28cf700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc19d3e0 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd28cf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc19d520 con 0x7f7fcc1006c0 2023-07-14T18:10:21.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd28cf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc19d6b0 con 0x7f7fcc0feb40 2023-07-14T18:10:21.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0ffd30 0x7f7fcc195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:21.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0ffd30 0x7f7fcc195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52574/0 (socket says 172.21.15.115:52574) 2023-07-14T18:10:21.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc3fff700 1 -- 172.21.15.115:0/3647523505 learned_addr learned my addr 172.21.15.115:0/3647523505 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:21.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd0e6c700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fcc1006c0 0x7f7fcc19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:21.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc3fff700 1 -- 172.21.15.115:0/3647523505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fcc0feb40 msgr2=0x7f7fcc195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:21.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fcbfff700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fcc0feb40 0x7f7fcc195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:21.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc3fff700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fcc0feb40 0x7f7fcc195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc3fff700 1 -- 172.21.15.115:0/3647523505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fcc1006c0 msgr2=0x7f7fcc19ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:21.629 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc3fff700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fcc1006c0 0x7f7fcc19ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:21.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc3fff700 1 -- 172.21.15.115:0/3647523505 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc0095f0 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fcbfff700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fcc0feb40 0x7f7fcc195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:21.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc3fff700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0ffd30 0x7f7fcc195da0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c9b0 tx=0x7f7fb400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:21.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc9ffb700 1 -- 172.21.15.115:0/3647523505 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4007b00 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc9ffb700 1 -- 172.21.15.115:0/3647523505 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7fb4007c60 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc19d990 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fc9ffb700 1 -- 172.21.15.115:0/3647523505 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4018660 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.630 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.586+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc19def0 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.587+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc10a210 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.589+0000 7f7fc9ffb700 1 -- 172.21.15.115:0/3647523505 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7fb400f450 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.589+0000 7f7fc9ffb700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7fb80a11d0 0x7f7fb80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:21.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.589+0000 7f7fcbfff700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7fb80a11d0 0x7f7fb80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:21.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.589+0000 7f7fc9ffb700 1 -- 172.21.15.115:0/3647523505 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f7fb40ccd20 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.590+0000 7f7fcbfff700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7fb80a11d0 0x7f7fb80a3680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0098e0 tx=0x7f7fbc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:21.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.593+0000 7f7fc9ffb700 1 -- 172.21.15.115:0/3647523505 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7fb40963b0 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:21.781+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f7fcc10a3d0 con 0x7f7fcc0ffd30 2023-07-14T18:10:21.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:21 smithi115 ceph-mon[127451]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T18:10:21.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:21 smithi115 ceph-mon[127451]: pgmap v250: 97 pgs: 42 active+clean, 28 creating+peering, 27 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2023-07-14T18:10:21.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:21 smithi150 ceph-mon[128853]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T18:10:21.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:21 smithi150 ceph-mon[128853]: pgmap v250: 97 pgs: 42 active+clean, 28 creating+peering, 27 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2023-07-14T18:10:21.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:21 smithi187 ceph-mon[129317]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T18:10:21.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:21 smithi187 ceph-mon[129317]: pgmap v250: 97 pgs: 42 active+clean, 28 creating+peering, 27 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2023-07-14T18:10:22.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.437+0000 7f7fc9ffb700 1 -- 172.21.15.115:0/3647523505 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7fb400fe60 con 0x7f7fcc0ffd30 2023-07-14T18:10:22.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.443+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7fb80a11d0 msgr2=0x7f7fb80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:22.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.443+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7fb80a11d0 0x7f7fb80a3680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0098e0 tx=0x7f7fbc005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.443+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0ffd30 msgr2=0x7f7fcc195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:22.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.443+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0ffd30 0x7f7fcc195da0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400c9b0 tx=0x7f7fb400ccc0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.448+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 shutdown_connections 2023-07-14T18:10:22.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.448+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7fb80a11d0 0x7f7fb80a3680 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.448+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fcc1006c0 0x7f7fcc19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.448+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7fcc0feb40 0x7f7fcc195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.448+0000 7f7fd28cf700 1 --2- 172.21.15.115:0/3647523505 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7fcc0ffd30 0x7f7fcc195da0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.448+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 >> 172.21.15.115:0/3647523505 conn(0x7f7fcc0fa160 msgr2=0x7f7fcc068790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:22.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.448+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 shutdown_connections 2023-07-14T18:10:22.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.449+0000 7f7fd28cf700 1 -- 172.21.15.115:0/3647523505 wait complete. 2023-07-14T18:10:22.493 INFO:teuthology.orchestra.run.smithi115.stderr:set pool 4 allow_ec_overwrites to true 2023-07-14T18:10:22.509 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-07-14T18:10:22.620 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:10:22 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-b[154137]: 2023-07-14T18:10:22.182+0000 7f267e4fa700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-14T18:10:22.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:22 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2937769424' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T18:10:22.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:22 smithi150 ceph-mon[128853]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T18:10:22.621 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:22 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3647523505' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T18:10:22.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.598+0000 7f96c60fa700 1 -- 172.21.15.115:0/2260324349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c0100690 msgr2=0x7f96c0106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:22.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.598+0000 7f96c60fa700 1 --2- 172.21.15.115:0/2260324349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c0100690 0x7f96c0106f50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f96b800b0a0 tx=0x7f96b800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.598+0000 7f96c60fa700 1 -- 172.21.15.115:0/2260324349 shutdown_connections 2023-07-14T18:10:22.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.598+0000 7f96c60fa700 1 --2- 172.21.15.115:0/2260324349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c0100690 0x7f96c0106f50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.598+0000 7f96c60fa700 1 --2- 172.21.15.115:0/2260324349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96c00feb10 0x7f96c00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.598+0000 7f96c60fa700 1 --2- 172.21.15.115:0/2260324349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96c00ffd00 0x7f96c0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.598+0000 7f96c60fa700 1 -- 172.21.15.115:0/2260324349 >> 172.21.15.115:0/2260324349 conn(0x7f96c00fa130 msgr2=0x7f96c00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:22.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.598+0000 7f96c60fa700 1 -- 172.21.15.115:0/2260324349 shutdown_connections 2023-07-14T18:10:22.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.598+0000 7f96c60fa700 1 -- 172.21.15.115:0/2260324349 wait complete. 2023-07-14T18:10:22.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96c60fa700 1 Processor -- start 2023-07-14T18:10:22.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96c60fa700 1 -- start start 2023-07-14T18:10:22.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96c60fa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c00feb10 0x7f96c0195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:22.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96c60fa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96c00ffd00 0x7f96c0195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:22.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96c60fa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96c0100690 0x7f96c019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:22.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96c60fa700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c019d5a0 con 0x7f96c00feb10 2023-07-14T18:10:22.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96c60fa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c019d6e0 con 0x7f96c00ffd00 2023-07-14T18:10:22.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96c60fa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c019d870 con 0x7f96c0100690 2023-07-14T18:10:22.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bf7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c00feb10 0x7f96c0195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:22.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bf7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c00feb10 0x7f96c0195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52590/0 (socket says 172.21.15.115:52590) 2023-07-14T18:10:22.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bf7fe700 1 -- 172.21.15.115:0/983100408 learned_addr learned my addr 172.21.15.115:0/983100408 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:22.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96beffd700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96c00ffd00 0x7f96c0195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:22.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bffff700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96c0100690 0x7f96c019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:22.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bf7fe700 1 -- 172.21.15.115:0/983100408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96c0100690 msgr2=0x7f96c019cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:22.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bf7fe700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96c0100690 0x7f96c019cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bf7fe700 1 -- 172.21.15.115:0/983100408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96c00ffd00 msgr2=0x7f96c0195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:22.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bf7fe700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96c00ffd00 0x7f96c0195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:22.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bf7fe700 1 -- 172.21.15.115:0/983100408 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b0009520 con 0x7f96c00feb10 2023-07-14T18:10:22.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.599+0000 7f96bffff700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96c0100690 0x7f96c019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:22.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.600+0000 7f96bf7fe700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c00feb10 0x7f96c0195a20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f96a800e970 tx=0x7f96a800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:22.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.600+0000 7f96bcff9700 1 -- 172.21.15.115:0/983100408 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8009790 con 0x7f96c00feb10 2023-07-14T18:10:22.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.600+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b8009d00 con 0x7f96c00feb10 2023-07-14T18:10:22.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.600+0000 7f96bcff9700 1 -- 172.21.15.115:0/983100408 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f96a800eec0 con 0x7f96c00feb10 2023-07-14T18:10:22.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.600+0000 7f96bcff9700 1 -- 172.21.15.115:0/983100408 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8003c10 con 0x7f96c00feb10 2023-07-14T18:10:22.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.600+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c019df20 con 0x7f96c00feb10 2023-07-14T18:10:22.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.601+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c0106c90 con 0x7f96c00feb10 2023-07-14T18:10:22.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.602+0000 7f96bcff9700 1 -- 172.21.15.115:0/983100408 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f96a8003d90 con 0x7f96c00feb10 2023-07-14T18:10:22.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.602+0000 7f96bcff9700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f96ac0a1150 0x7f96ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:22.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.602+0000 7f96beffd700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f96ac0a1150 0x7f96ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:22.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.602+0000 7f96bcff9700 1 -- 172.21.15.115:0/983100408 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f96a8014070 con 0x7f96c00feb10 2023-07-14T18:10:22.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.603+0000 7f96beffd700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f96ac0a1150 0x7f96ac0a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009df0 tx=0x7f96b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:22.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.606+0000 7f96bcff9700 1 -- 172.21.15.115:0/983100408 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f96a8096440 con 0x7f96c00feb10 2023-07-14T18:10:22.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:22 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2937769424' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T18:10:22.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:22 smithi115 ceph-mon[127451]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T18:10:22.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:22 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3647523505' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T18:10:22.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:22.841+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f96c002c8e0 con 0x7f96c00feb10 2023-07-14T18:10:22.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:22 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2937769424' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T18:10:22.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:22 smithi187 ceph-mon[129317]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T18:10:22.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:22 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3647523505' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T18:10:23.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.440+0000 7f96bcff9700 1 -- 172.21.15.115:0/983100408 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f96a8099a60 con 0x7f96c00feb10 2023-07-14T18:10:23.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.443+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f96ac0a1150 msgr2=0x7f96ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:23.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.443+0000 7f96c60fa700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f96ac0a1150 0x7f96ac0a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009df0 tx=0x7f96b0009260 comp rx=0 tx=0).stop 2023-07-14T18:10:23.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.443+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c00feb10 msgr2=0x7f96c0195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:23.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.443+0000 7f96c60fa700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c00feb10 0x7f96c0195a20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f96a800e970 tx=0x7f96a800ec80 comp rx=0 tx=0).stop 2023-07-14T18:10:23.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.444+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 shutdown_connections 2023-07-14T18:10:23.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.444+0000 7f96c60fa700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f96ac0a1150 0x7f96ac0a3600 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.444+0000 7f96c60fa700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f96c0100690 0x7f96c019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.444+0000 7f96c60fa700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f96c00feb10 0x7f96c0195a20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.444+0000 7f96c60fa700 1 --2- 172.21.15.115:0/983100408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96c00ffd00 0x7f96c0195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.444+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 >> 172.21.15.115:0/983100408 conn(0x7f96c00fa130 msgr2=0x7f96c00fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:23.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.444+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 shutdown_connections 2023-07-14T18:10:23.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.444+0000 7f96c60fa700 1 -- 172.21.15.115:0/983100408 wait complete. 2023-07-14T18:10:23.483 INFO:teuthology.orchestra.run.smithi115.stderr:added data pool 4 to fsmap 2023-07-14T18:10:23.495 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:10:23.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.580+0000 7fad775fc700 1 -- 172.21.15.115:0/1180351815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 msgr2=0x7fad700fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:23.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.580+0000 7fad775fc700 1 --2- 172.21.15.115:0/1180351815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 0x7fad700fef10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fad64009990 tx=0x7fad64009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.580+0000 7fad775fc700 1 -- 172.21.15.115:0/1180351815 shutdown_connections 2023-07-14T18:10:23.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.580+0000 7fad775fc700 1 --2- 172.21.15.115:0/1180351815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad70100680 0x7fad70106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.580+0000 7fad775fc700 1 --2- 172.21.15.115:0/1180351815 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 0x7fad700fef10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.580+0000 7fad775fc700 1 --2- 172.21.15.115:0/1180351815 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad700ffcf0 0x7fad70100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.580+0000 7fad775fc700 1 -- 172.21.15.115:0/1180351815 >> 172.21.15.115:0/1180351815 conn(0x7fad700fa100 msgr2=0x7fad700fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:23.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.580+0000 7fad775fc700 1 -- 172.21.15.115:0/1180351815 shutdown_connections 2023-07-14T18:10:23.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.580+0000 7fad775fc700 1 -- 172.21.15.115:0/1180351815 wait complete. 2023-07-14T18:10:23.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad775fc700 1 Processor -- start 2023-07-14T18:10:23.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad775fc700 1 -- start start 2023-07-14T18:10:23.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad775fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 0x7fad70068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:23.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad775fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700ffcf0 0x7fad7019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:23.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad775fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad70100680 0x7fad7019cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:23.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad775fc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7019d430 con 0x7fad700feb00 2023-07-14T18:10:23.622 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad775fc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7019d5a0 con 0x7fad700ffcf0 2023-07-14T18:10:23.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad775fc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7019d760 con 0x7fad70100680 2023-07-14T18:10:23.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad75398700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 0x7fad70068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:23.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad75398700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 0x7fad70068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52604/0 (socket says 172.21.15.115:52604) 2023-07-14T18:10:23.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad75398700 1 -- 172.21.15.115:0/1699841192 learned_addr learned my addr 172.21.15.115:0/1699841192 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:23.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad74b97700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700ffcf0 0x7fad7019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:23.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.581+0000 7fad75b99700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad70100680 0x7fad7019cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:23.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad75398700 1 -- 172.21.15.115:0/1699841192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad70100680 msgr2=0x7fad7019cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:23.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad75398700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad70100680 0x7fad7019cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad75398700 1 -- 172.21.15.115:0/1699841192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700ffcf0 msgr2=0x7fad7019a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:23.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad75398700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700ffcf0 0x7fad7019a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad75398700 1 -- 172.21.15.115:0/1699841192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad640095f0 con 0x7fad700feb00 2023-07-14T18:10:23.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad75398700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 0x7fad70068330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fad64004730 tx=0x7fad64009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:23.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad627fc700 1 -- 172.21.15.115:0/1699841192 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6401d070 con 0x7fad700feb00 2023-07-14T18:10:23.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad627fc700 1 -- 172.21.15.115:0/1699841192 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fad64004ca0 con 0x7fad700feb00 2023-07-14T18:10:23.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7019d9e0 con 0x7fad700feb00 2023-07-14T18:10:23.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad627fc700 1 -- 172.21.15.115:0/1699841192 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64003af0 con 0x7fad700feb00 2023-07-14T18:10:23.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.582+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7019de60 con 0x7fad700feb00 2023-07-14T18:10:23.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.583+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad70105ae0 con 0x7fad700feb00 2023-07-14T18:10:23.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.584+0000 7fad627fc700 1 -- 172.21.15.115:0/1699841192 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fad64005000 con 0x7fad700feb00 2023-07-14T18:10:23.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.584+0000 7fad627fc700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad5c0a1220 0x7fad5c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:23.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.584+0000 7fad627fc700 1 -- 172.21.15.115:0/1699841192 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fad640cde10 con 0x7fad700feb00 2023-07-14T18:10:23.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.584+0000 7fad74b97700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad5c0a1220 0x7fad5c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:23.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.585+0000 7fad74b97700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad5c0a1220 0x7fad5c0a36d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad700680a0 tx=0x7fad6c00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:23.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.588+0000 7fad627fc700 1 -- 172.21.15.115:0/1699841192 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fad64097400 con 0x7fad700feb00 2023-07-14T18:10:23.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:23 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3647523505' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T18:10:23.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:23 smithi115 ceph-mon[127451]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T18:10:23.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:23 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/983100408' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T18:10:23.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:23 smithi115 ceph-mon[127451]: pgmap v253: 129 pgs: 101 active+clean, 15 creating+peering, 13 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2023-07-14T18:10:23.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.822+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad7002c8e0 con 0x7fad700feb00 2023-07-14T18:10:23.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.822+0000 7fad627fc700 1 -- 172.21.15.115:0/1699841192 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10790 (secure 0 0 0) 0x7fad6409aa20 con 0x7fad700feb00 2023-07-14T18:10:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:23.861 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:10:23.434369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-14T18:10:23.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad5c0a1220 msgr2=0x7fad5c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:23.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad5c0a1220 0x7fad5c0a36d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fad700680a0 tx=0x7fad6c00a300 comp rx=0 tx=0).stop 2023-07-14T18:10:23.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 msgr2=0x7fad70068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:23.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 0x7fad70068330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fad64004730 tx=0x7fad64009580 comp rx=0 tx=0).stop 2023-07-14T18:10:23.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 shutdown_connections 2023-07-14T18:10:23.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad5c0a1220 0x7fad5c0a36d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad70100680 0x7fad7019cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad700feb00 0x7fad70068330 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 --2- 172.21.15.115:0/1699841192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad700ffcf0 0x7fad7019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 >> 172.21.15.115:0/1699841192 conn(0x7fad700fa100 msgr2=0x7fad700fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:23.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 shutdown_connections 2023-07-14T18:10:23.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.825+0000 7fad775fc700 1 -- 172.21.15.115:0/1699841192 wait complete. 2023-07-14T18:10:23.865 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 17 2023-07-14T18:10:23.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:23 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3647523505' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T18:10:23.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:23 smithi150 ceph-mon[128853]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T18:10:23.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:23 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/983100408' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T18:10:23.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:23 smithi150 ceph-mon[128853]: pgmap v253: 129 pgs: 101 active+clean, 15 creating+peering, 13 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2023-07-14T18:10:23.875 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:10:23.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:23 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3647523505' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T18:10:23.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:23 smithi187 ceph-mon[129317]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T18:10:23.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:23 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/983100408' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T18:10:23.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:23 smithi187 ceph-mon[129317]: pgmap v253: 129 pgs: 101 active+clean, 15 creating+peering, 13 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2023-07-14T18:10:23.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.958+0000 7f52ece16700 1 -- 172.21.15.115:0/1578769101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 msgr2=0x7f52e80ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:23.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.958+0000 7f52ece16700 1 --2- 172.21.15.115:0/1578769101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 0x7f52e80ff1b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009990 tx=0x7f52d8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.958+0000 7f52ece16700 1 -- 172.21.15.115:0/1578769101 shutdown_connections 2023-07-14T18:10:23.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.958+0000 7f52ece16700 1 --2- 172.21.15.115:0/1578769101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e80ff6f0 0x7f52e8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.958+0000 7f52ece16700 1 --2- 172.21.15.115:0/1578769101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e8100180 0x7f52e8100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.997 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.958+0000 7f52ece16700 1 --2- 172.21.15.115:0/1578769101 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 0x7f52e80ff1b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:23.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.958+0000 7f52ece16700 1 -- 172.21.15.115:0/1578769101 >> 172.21.15.115:0/1578769101 conn(0x7f52e80f98a0 msgr2=0x7f52e80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:23.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.958+0000 7f52ece16700 1 -- 172.21.15.115:0/1578769101 shutdown_connections 2023-07-14T18:10:23.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.958+0000 7f52ece16700 1 -- 172.21.15.115:0/1578769101 wait complete. 2023-07-14T18:10:24.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52ece16700 1 Processor -- start 2023-07-14T18:10:24.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52ece16700 1 -- start start 2023-07-14T18:10:24.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52ece16700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 0x7f52e8195760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52ece16700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80ff6f0 0x7f52e8195ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52ece16700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e8100180 0x7f52e819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52ece16700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e819d2e0 con 0x7f52e80fed40 2023-07-14T18:10:24.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52ece16700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e819d420 con 0x7f52e8100180 2023-07-14T18:10:24.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52ece16700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e819d5b0 con 0x7f52e80ff6f0 2023-07-14T18:10:24.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 0x7f52e8195760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 0x7f52e8195760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52618/0 (socket says 172.21.15.115:52618) 2023-07-14T18:10:24.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e659c700 1 -- 172.21.15.115:0/4077186950 learned_addr learned my addr 172.21.15.115:0/4077186950 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:24.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e5d9b700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80ff6f0 0x7f52e8195ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e6d9d700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e8100180 0x7f52e819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e659c700 1 -- 172.21.15.115:0/4077186950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80ff6f0 msgr2=0x7f52e8195ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e659c700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80ff6f0 0x7f52e8195ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e659c700 1 -- 172.21.15.115:0/4077186950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e8100180 msgr2=0x7f52e819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e659c700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e8100180 0x7f52e819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e659c700 1 -- 172.21.15.115:0/4077186950 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d80095f0 con 0x7f52e80fed40 2023-07-14T18:10:24.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e6d9d700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e8100180 0x7f52e819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:24.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.959+0000 7f52e659c700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 0x7f52e8195760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00d950 tx=0x7f52dc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:24.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.960+0000 7f52d77fe700 1 -- 172.21.15.115:0/4077186950 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc009b40 con 0x7f52e80fed40 2023-07-14T18:10:24.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.960+0000 7f52d77fe700 1 -- 172.21.15.115:0/4077186950 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f52dc009ca0 con 0x7f52e80fed40 2023-07-14T18:10:24.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.960+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e819d890 con 0x7f52e80fed40 2023-07-14T18:10:24.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.960+0000 7f52d77fe700 1 -- 172.21.15.115:0/4077186950 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc00f620 con 0x7f52e80fed40 2023-07-14T18:10:24.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.960+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e819ddf0 con 0x7f52e80fed40 2023-07-14T18:10:24.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.961+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e810a180 con 0x7f52e80fed40 2023-07-14T18:10:24.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.962+0000 7f52d77fe700 1 -- 172.21.15.115:0/4077186950 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f52dc010460 con 0x7f52e80fed40 2023-07-14T18:10:24.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.962+0000 7f52d77fe700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52d00a1220 0x7f52d00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.962+0000 7f52e5d9b700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52d00a1220 0x7f52d00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.962+0000 7f52d77fe700 1 -- 172.21.15.115:0/4077186950 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f52dc0ccc40 con 0x7f52e80fed40 2023-07-14T18:10:24.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.963+0000 7f52e5d9b700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52d00a1220 0x7f52d00a36d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f52d800b5c0 tx=0x7f52d8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:24.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:23.965+0000 7f52d77fe700 1 -- 172.21.15.115:0/4077186950 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f52dc0961a0 con 0x7f52e80fed40 2023-07-14T18:10:24.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.157+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52e810a370 con 0x7f52e80fed40 2023-07-14T18:10:24.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.158+0000 7f52d77fe700 1 -- 172.21.15.115:0/4077186950 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7f52dc020090 con 0x7f52e80fed40 2023-07-14T18:10:24.197 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:24.197 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:24.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.160+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52d00a1220 msgr2=0x7f52d00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.160+0000 7f52ece16700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52d00a1220 0x7f52d00a36d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f52d800b5c0 tx=0x7f52d8005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.160+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 msgr2=0x7f52e8195760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.160+0000 7f52ece16700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 0x7f52e8195760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00d950 tx=0x7f52dc00dd10 comp rx=0 tx=0).stop 2023-07-14T18:10:24.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.161+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 shutdown_connections 2023-07-14T18:10:24.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.161+0000 7f52ece16700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52d00a1220 0x7f52d00a36d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.161+0000 7f52ece16700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e8100180 0x7f52e819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.161+0000 7f52ece16700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52e80fed40 0x7f52e8195760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.161+0000 7f52ece16700 1 --2- 172.21.15.115:0/4077186950 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80ff6f0 0x7f52e8195ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.161+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 >> 172.21.15.115:0/4077186950 conn(0x7f52e80f98a0 msgr2=0x7f52e80fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:24.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.161+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 shutdown_connections 2023-07-14T18:10:24.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.161+0000 7f52ece16700 1 -- 172.21.15.115:0/4077186950 wait complete. 2023-07-14T18:10:24.212 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:10:24.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.295+0000 7fe4bd746700 1 -- 172.21.15.115:0/2934851678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 msgr2=0x7fe4b8106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.295+0000 7fe4bd746700 1 --2- 172.21.15.115:0/2934851678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 0x7fe4b8106f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000b0a0 tx=0x7fe4b000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 -- 172.21.15.115:0/2934851678 shutdown_connections 2023-07-14T18:10:24.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 --2- 172.21.15.115:0/2934851678 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 0x7fe4b8106f90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 --2- 172.21.15.115:0/2934851678 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b80feb50 0x7fe4b80fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 --2- 172.21.15.115:0/2934851678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4b80ffd40 0x7fe4b8100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 -- 172.21.15.115:0/2934851678 >> 172.21.15.115:0/2934851678 conn(0x7fe4b80fa150 msgr2=0x7fe4b80fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:24.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 -- 172.21.15.115:0/2934851678 shutdown_connections 2023-07-14T18:10:24.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 -- 172.21.15.115:0/2934851678 wait complete. 2023-07-14T18:10:24.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 Processor -- start 2023-07-14T18:10:24.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 -- start start 2023-07-14T18:10:24.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4b80feb50 0x7fe4b8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b80ffd40 0x7fe4b819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.296+0000 7fe4bd746700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 0x7fe4b819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4bd746700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b819d450 con 0x7fe4b81006d0 2023-07-14T18:10:24.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4bd746700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b819d5c0 con 0x7fe4b80feb50 2023-07-14T18:10:24.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4bd746700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b819d780 con 0x7fe4b80ffd40 2023-07-14T18:10:24.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 0x7fe4b819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 0x7fe4b819cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52636/0 (socket says 172.21.15.115:52636) 2023-07-14T18:10:24.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b77fe700 1 -- 172.21.15.115:0/512418092 learned_addr learned my addr 172.21.15.115:0/512418092 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:24.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b67fc700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b80ffd40 0x7fe4b819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b6ffd700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4b80feb50 0x7fe4b8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b77fe700 1 -- 172.21.15.115:0/512418092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b80ffd40 msgr2=0x7fe4b819a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b77fe700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b80ffd40 0x7fe4b819a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b77fe700 1 -- 172.21.15.115:0/512418092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4b80feb50 msgr2=0x7fe4b8069590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b77fe700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4b80feb50 0x7fe4b8069590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b77fe700 1 -- 172.21.15.115:0/512418092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b0009d00 con 0x7fe4b81006d0 2023-07-14T18:10:24.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4b77fe700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 0x7fe4b819cc80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000bad0 tx=0x7fe4b000bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:24.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe49ffff700 1 -- 172.21.15.115:0/512418092 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b000e040 con 0x7fe4b81006d0 2023-07-14T18:10:24.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe49ffff700 1 -- 172.21.15.115:0/512418092 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe4b0004e40 con 0x7fe4b81006d0 2023-07-14T18:10:24.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b819da00 con 0x7fe4b81006d0 2023-07-14T18:10:24.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe49ffff700 1 -- 172.21.15.115:0/512418092 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b00052a0 con 0x7fe4b81006d0 2023-07-14T18:10:24.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.297+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b819de80 con 0x7fe4b81006d0 2023-07-14T18:10:24.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.298+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b8105ff0 con 0x7fe4b81006d0 2023-07-14T18:10:24.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.299+0000 7fe49ffff700 1 -- 172.21.15.115:0/512418092 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fe4b001d990 con 0x7fe4b81006d0 2023-07-14T18:10:24.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.300+0000 7fe49ffff700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe4a00a11a0 0x7fe4a00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.300+0000 7fe4b6ffd700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe4a00a11a0 0x7fe4a00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.300+0000 7fe49ffff700 1 -- 172.21.15.115:0/512418092 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe4b00ce1d0 con 0x7fe4b81006d0 2023-07-14T18:10:24.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.300+0000 7fe4b6ffd700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe4a00a11a0 0x7fe4a00a3650 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8005950 tx=0x7fe4a80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:24.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.304+0000 7fe49ffff700 1 -- 172.21.15.115:0/512418092 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe4b0097760 con 0x7fe4b81006d0 2023-07-14T18:10:24.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.495+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4b80612f0 con 0x7fe4b81006d0 2023-07-14T18:10:24.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.496+0000 7fe49ffff700 1 -- 172.21.15.115:0/512418092 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7fe4b009ad80 con 0x7fe4b81006d0 2023-07-14T18:10:24.535 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:24.535 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:24.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe4a00a11a0 msgr2=0x7fe4a00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe4a00a11a0 0x7fe4a00a3650 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8005950 tx=0x7fe4a80058e0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 msgr2=0x7fe4b819cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 0x7fe4b819cc80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000bad0 tx=0x7fe4b000bb00 comp rx=0 tx=0).stop 2023-07-14T18:10:24.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 shutdown_connections 2023-07-14T18:10:24.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe4a00a11a0 0x7fe4a00a3650 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe4b81006d0 0x7fe4b819cc80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4b80feb50 0x7fe4b8069590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 --2- 172.21.15.115:0/512418092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4b80ffd40 0x7fe4b819a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 >> 172.21.15.115:0/512418092 conn(0x7fe4b80fa150 msgr2=0x7fe4b8103130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:24.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 shutdown_connections 2023-07-14T18:10:24.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.499+0000 7fe4bd746700 1 -- 172.21.15.115:0/512418092 wait complete. 2023-07-14T18:10:24.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:10:24.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.648+0000 7ffb87788700 1 -- 172.21.15.115:0/2229306000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb800fed00 msgr2=0x7ffb800ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.648+0000 7ffb87788700 1 --2- 172.21.15.115:0/2229306000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb800fed00 0x7ffb800ff170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009990 tx=0x7ffb7c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.648+0000 7ffb87788700 1 -- 172.21.15.115:0/2229306000 shutdown_connections 2023-07-14T18:10:24.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.648+0000 7ffb87788700 1 --2- 172.21.15.115:0/2229306000 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb800ff6b0 0x7ffb80106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.648+0000 7ffb87788700 1 --2- 172.21.15.115:0/2229306000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb801000b0 0x7ffb801004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.648+0000 7ffb87788700 1 --2- 172.21.15.115:0/2229306000 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb800fed00 0x7ffb800ff170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.648+0000 7ffb87788700 1 -- 172.21.15.115:0/2229306000 >> 172.21.15.115:0/2229306000 conn(0x7ffb800f98a0 msgr2=0x7ffb800fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:24.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.648+0000 7ffb87788700 1 -- 172.21.15.115:0/2229306000 shutdown_connections 2023-07-14T18:10:24.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.648+0000 7ffb87788700 1 -- 172.21.15.115:0/2229306000 wait complete. 2023-07-14T18:10:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb87788700 1 Processor -- start 2023-07-14T18:10:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb87788700 1 -- start start 2023-07-14T18:10:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb87788700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb800fed00 0x7ffb80194880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb87788700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb800ff6b0 0x7ffb80194dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb87788700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb801000b0 0x7ffb8019be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb87788700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019c400 con 0x7ffb801000b0 2023-07-14T18:10:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb87788700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019c540 con 0x7ffb800ff6b0 2023-07-14T18:10:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb87788700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019c6d0 con 0x7ffb800fed00 2023-07-14T18:10:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85d25700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb801000b0 0x7ffb8019be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85d25700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb801000b0 0x7ffb8019be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52650/0 (socket says 172.21.15.115:52650) 2023-07-14T18:10:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85d25700 1 -- 172.21.15.115:0/3914532483 learned_addr learned my addr 172.21.15.115:0/3914532483 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb84d23700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb800ff6b0 0x7ffb80194dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85524700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb800fed00 0x7ffb80194880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85d25700 1 -- 172.21.15.115:0/3914532483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb800fed00 msgr2=0x7ffb80194880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85d25700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb800fed00 0x7ffb80194880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85d25700 1 -- 172.21.15.115:0/3914532483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb800ff6b0 msgr2=0x7ffb80194dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85d25700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb800ff6b0 0x7ffb80194dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85d25700 1 -- 172.21.15.115:0/3914532483 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7c0095f0 con 0x7ffb801000b0 2023-07-14T18:10:24.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85524700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb800fed00 0x7ffb80194880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:24.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.649+0000 7ffb85d25700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb801000b0 0x7ffb8019be30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800cfb0 tx=0x7ffb7800e330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:24.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.650+0000 7ffb767fc700 1 -- 172.21.15.115:0/3914532483 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78011070 con 0x7ffb801000b0 2023-07-14T18:10:24.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.650+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb800fdc50 con 0x7ffb801000b0 2023-07-14T18:10:24.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.650+0000 7ffb767fc700 1 -- 172.21.15.115:0/3914532483 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ffb78007b30 con 0x7ffb801000b0 2023-07-14T18:10:24.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.650+0000 7ffb767fc700 1 -- 172.21.15.115:0/3914532483 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7800ad90 con 0x7ffb801000b0 2023-07-14T18:10:24.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.650+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb800fe110 con 0x7ffb801000b0 2023-07-14T18:10:24.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.651+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8010a230 con 0x7ffb801000b0 2023-07-14T18:10:24.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.652+0000 7ffb767fc700 1 -- 172.21.15.115:0/3914532483 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ffb7800ea40 con 0x7ffb801000b0 2023-07-14T18:10:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.653+0000 7ffb767fc700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.653+0000 7ffb85524700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:24.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.653+0000 7ffb767fc700 1 -- 172.21.15.115:0/3914532483 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffb780ced70 con 0x7ffb801000b0 2023-07-14T18:10:24.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.653+0000 7ffb85524700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffb70007900 tx=0x7ffb70008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:24.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.656+0000 7ffb767fc700 1 -- 172.21.15.115:0/3914532483 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ffb780982d0 con 0x7ffb801000b0 2023-07-14T18:10:24.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:24 smithi115 ceph-mon[127451]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T18:10:24.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:24 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/983100408' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T18:10:24.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:24 smithi115 ceph-mon[127451]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:24.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:24 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1699841192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:24.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:24 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4077186950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:24.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:24 smithi150 ceph-mon[128853]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T18:10:24.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:24 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/983100408' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T18:10:24.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:24 smithi150 ceph-mon[128853]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:24.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:24 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1699841192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:24.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:24 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/4077186950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:24.926 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.888+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb800fea00 con 0x7ffb801000b0 2023-07-14T18:10:24.927 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.889+0000 7ffb767fc700 1 -- 172.21.15.115:0/3914532483 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10790 (secure 0 0 0) 0x7ffb7809b8f0 con 0x7ffb801000b0 2023-07-14T18:10:24.927 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:24.927 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:10:23.434369+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-14T18:10:24.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ffb6c0a11a0 msgr2=0x7ffb6c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffb70007900 tx=0x7ffb70008040 comp rx=0 tx=0).stop 2023-07-14T18:10:24.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb801000b0 msgr2=0x7ffb8019be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:24.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb801000b0 0x7ffb8019be30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800cfb0 tx=0x7ffb7800e330 comp rx=0 tx=0).stop 2023-07-14T18:10:24.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 shutdown_connections 2023-07-14T18:10:24.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ffb801000b0 0x7ffb8019be30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb800fed00 0x7ffb80194880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 --2- 172.21.15.115:0/3914532483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb800ff6b0 0x7ffb80194dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:24.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 >> 172.21.15.115:0/3914532483 conn(0x7ffb800f98a0 msgr2=0x7ffb800fbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:24.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 shutdown_connections 2023-07-14T18:10:24.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:24.891+0000 7ffb87788700 1 -- 172.21.15.115:0/3914532483 wait complete. 2023-07-14T18:10:24.931 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 17 2023-07-14T18:10:24.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:24 smithi187 ceph-mon[129317]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T18:10:24.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:24 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/983100408' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T18:10:24.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:24 smithi187 ceph-mon[129317]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:24.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:24 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1699841192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:24.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:24 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/4077186950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:24.942 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:10:25.062 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 -- 172.21.15.115:0/1162770808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40ffed0 msgr2=0x7f41a4106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 --2- 172.21.15.115:0/1162770808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40ffed0 0x7f41a4106790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f419c00c210 tx=0x7f419c00c520 comp rx=0 tx=0).stop 2023-07-14T18:10:25.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 -- 172.21.15.115:0/1162770808 shutdown_connections 2023-07-14T18:10:25.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 --2- 172.21.15.115:0/1162770808 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40ffed0 0x7f41a4106790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 --2- 172.21.15.115:0/1162770808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40fe350 0x7f41a40fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 --2- 172.21.15.115:0/1162770808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a40ff540 0x7f41a40ff990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 -- 172.21.15.115:0/1162770808 >> 172.21.15.115:0/1162770808 conn(0x7f41a40f9990 msgr2=0x7f41a40fbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:25.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 -- 172.21.15.115:0/1162770808 shutdown_connections 2023-07-14T18:10:25.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 -- 172.21.15.115:0/1162770808 wait complete. 2023-07-14T18:10:25.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.024+0000 7f41a9981700 1 Processor -- start 2023-07-14T18:10:25.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a9981700 1 -- start start 2023-07-14T18:10:25.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a9981700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40fe350 0x7f41a4195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a9981700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40ff540 0x7f41a4195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a9981700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a40ffed0 0x7f41a419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a9981700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a419d4f0 con 0x7f41a40fe350 2023-07-14T18:10:25.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a9981700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a419d630 con 0x7f41a40ff540 2023-07-14T18:10:25.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a9981700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a419d7c0 con 0x7f41a40ffed0 2023-07-14T18:10:25.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a2ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40fe350 0x7f41a4195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a2ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40fe350 0x7f41a4195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52660/0 (socket says 172.21.15.115:52660) 2023-07-14T18:10:25.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a2ffd700 1 -- 172.21.15.115:0/991634724 learned_addr learned my addr 172.21.15.115:0/991634724 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:25.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a27fc700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40ff540 0x7f41a4195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a37fe700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a40ffed0 0x7f41a419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a2ffd700 1 -- 172.21.15.115:0/991634724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a40ffed0 msgr2=0x7f41a419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a2ffd700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a40ffed0 0x7f41a419cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a2ffd700 1 -- 172.21.15.115:0/991634724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40ff540 msgr2=0x7f41a4195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a2ffd700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40ff540 0x7f41a4195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a2ffd700 1 -- 172.21.15.115:0/991634724 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c00ae10 con 0x7f41a40fe350 2023-07-14T18:10:25.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a37fe700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a40ffed0 0x7f41a419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:25.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a27fc700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40ff540 0x7f41a4195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:25.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a2ffd700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40fe350 0x7f41a4195970 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f419400c970 tx=0x7f419400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:25.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.025+0000 7f41a897f700 1 -- 172.21.15.115:0/991634724 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194004220 con 0x7f41a40fe350 2023-07-14T18:10:25.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.026+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a419daa0 con 0x7f41a40fe350 2023-07-14T18:10:25.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.026+0000 7f41a897f700 1 -- 172.21.15.115:0/991634724 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4194007d30 con 0x7f41a40fe350 2023-07-14T18:10:25.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.026+0000 7f41a897f700 1 -- 172.21.15.115:0/991634724 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194003bf0 con 0x7f41a40fe350 2023-07-14T18:10:25.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.026+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a419dfb0 con 0x7f41a40fe350 2023-07-14T18:10:25.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.026+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a4105840 con 0x7f41a40fe350 2023-07-14T18:10:25.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.028+0000 7f41a897f700 1 -- 172.21.15.115:0/991634724 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f4194003d70 con 0x7f41a40fe350 2023-07-14T18:10:25.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.028+0000 7f41a897f700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f418c0a11a0 0x7f418c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.028+0000 7f41a27fc700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f418c0a11a0 0x7f418c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.028+0000 7f41a897f700 1 -- 172.21.15.115:0/991634724 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f41940ccba0 con 0x7f41a40fe350 2023-07-14T18:10:25.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.029+0000 7f41a27fc700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f418c0a11a0 0x7f418c0a3650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f41980099d0 tx=0x7f4198008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:25.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.031+0000 7f41a897f700 1 -- 172.21.15.115:0/991634724 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4194096100 con 0x7f41a40fe350 2023-07-14T18:10:25.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.222+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41a402cb90 con 0x7f41a40fe350 2023-07-14T18:10:25.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.223+0000 7f41a897f700 1 -- 172.21.15.115:0/991634724 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7f419401d090 con 0x7f41a40fe350 2023-07-14T18:10:25.262 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:25.262 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:25.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.225+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f418c0a11a0 msgr2=0x7f418c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.225+0000 7f41a9981700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f418c0a11a0 0x7f418c0a3650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f41980099d0 tx=0x7f4198008040 comp rx=0 tx=0).stop 2023-07-14T18:10:25.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40fe350 msgr2=0x7f41a4195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40fe350 0x7f41a4195970 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f419400c970 tx=0x7f419400cd30 comp rx=0 tx=0).stop 2023-07-14T18:10:25.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 shutdown_connections 2023-07-14T18:10:25.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f418c0a11a0 0x7f418c0a3650 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a40ffed0 0x7f41a419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41a40fe350 0x7f41a4195970 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 --2- 172.21.15.115:0/991634724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40ff540 0x7f41a4195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 >> 172.21.15.115:0/991634724 conn(0x7f41a40f9990 msgr2=0x7f41a4102980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:25.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 shutdown_connections 2023-07-14T18:10:25.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.226+0000 7f41a9981700 1 -- 172.21.15.115:0/991634724 wait complete. 2023-07-14T18:10:25.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-07-14T18:10:17.634244+0000', 'modified': '2023-07-14T18:10:23.434369+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24515}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6839/2895627668', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 2895627668}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 2895627668}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-07-14T18:10:25.277 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-07-14T18:10:25.277 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T18:10:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.361+0000 7f59c54f6700 1 -- 172.21.15.115:0/3780600520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00ffd60 msgr2=0x7f59c01001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.361+0000 7f59c54f6700 1 --2- 172.21.15.115:0/3780600520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00ffd60 0x7f59c01001b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009990 tx=0x7f59b4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.361+0000 7f59c54f6700 1 -- 172.21.15.115:0/3780600520 shutdown_connections 2023-07-14T18:10:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.361+0000 7f59c54f6700 1 --2- 172.21.15.115:0/3780600520 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59c01006f0 0x7f59c0106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.361+0000 7f59c54f6700 1 --2- 172.21.15.115:0/3780600520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c00feb70 0x7f59c00fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.361+0000 7f59c54f6700 1 --2- 172.21.15.115:0/3780600520 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00ffd60 0x7f59c01001b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.361+0000 7f59c54f6700 1 -- 172.21.15.115:0/3780600520 >> 172.21.15.115:0/3780600520 conn(0x7f59c00fa1d0 msgr2=0x7f59c00fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.361+0000 7f59c54f6700 1 -- 172.21.15.115:0/3780600520 shutdown_connections 2023-07-14T18:10:25.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.361+0000 7f59c54f6700 1 -- 172.21.15.115:0/3780600520 wait complete. 2023-07-14T18:10:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59c54f6700 1 Processor -- start 2023-07-14T18:10:25.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59c54f6700 1 -- start start 2023-07-14T18:10:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59c54f6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00feb70 0x7f59c01958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59c54f6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c00ffd60 0x7f59c0195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59c54f6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59c01006f0 0x7f59c019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59c54f6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c019d440 con 0x7f59c00feb70 2023-07-14T18:10:25.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59c54f6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c019d580 con 0x7f59c00ffd60 2023-07-14T18:10:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59c54f6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c019d710 con 0x7f59c01006f0 2023-07-14T18:10:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59beffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00feb70 0x7f59c01958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59beffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00feb70 0x7f59c01958c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52674/0 (socket says 172.21.15.115:52674) 2023-07-14T18:10:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59beffd700 1 -- 172.21.15.115:0/4113773384 learned_addr learned my addr 172.21.15.115:0/4113773384 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59bf7fe700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59c01006f0 0x7f59c019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59beffd700 1 -- 172.21.15.115:0/4113773384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59c01006f0 msgr2=0x7f59c019ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59beffd700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59c01006f0 0x7f59c019ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59beffd700 1 -- 172.21.15.115:0/4113773384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c00ffd60 msgr2=0x7f59c0195e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59be7fc700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c00ffd60 0x7f59c0195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.362+0000 7f59beffd700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c00ffd60 0x7f59c0195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.363+0000 7f59beffd700 1 -- 172.21.15.115:0/4113773384 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40095f0 con 0x7f59c00feb70 2023-07-14T18:10:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.363+0000 7f59beffd700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00feb70 0x7f59c01958c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59b000d980 tx=0x7f59b000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.363+0000 7f59a7fff700 1 -- 172.21.15.115:0/4113773384 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b000de60 con 0x7f59c00feb70 2023-07-14T18:10:25.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.363+0000 7f59c54f6700 1 -- 172.21.15.115:0/4113773384 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c019d9f0 con 0x7f59c00feb70 2023-07-14T18:10:25.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.363+0000 7f59a7fff700 1 -- 172.21.15.115:0/4113773384 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f59b00044b0 con 0x7f59c00feb70 2023-07-14T18:10:25.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.363+0000 7f59a7fff700 1 -- 172.21.15.115:0/4113773384 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b0003b40 con 0x7f59c00feb70 2023-07-14T18:10:25.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.363+0000 7f59c54f6700 1 -- 172.21.15.115:0/4113773384 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c019df00 con 0x7f59c00feb70 2023-07-14T18:10:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.365+0000 7f59a7fff700 1 -- 172.21.15.115:0/4113773384 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f59b0003ca0 con 0x7f59c00feb70 2023-07-14T18:10:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.365+0000 7f59a7fff700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59a80a1150 0x7f59a80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.365+0000 7f59be7fc700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59a80a1150 0x7f59a80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.365+0000 7f59a7fff700 1 -- 172.21.15.115:0/4113773384 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f59b00cc810 con 0x7f59c00feb70 2023-07-14T18:10:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.366+0000 7f59be7fc700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59a80a1150 0x7f59a80a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f59b4000c00 tx=0x7f59b4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.366+0000 7f59d168eb80 1 Processor -- start 2023-07-14T18:10:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.366+0000 7f59d168eb80 1 -- start start 2023-07-14T18:10:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.366+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fa2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.366+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa2860 0x55bc62fa2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa3460 0x55bc62fa38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59d168eb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc62ea6c60 con 0x55bc62fa1f10 2023-07-14T18:10:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59d168eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc62ef99f0 con 0x55bc62fa2860 2023-07-14T18:10:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59d168eb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc62ef2e80 con 0x55bc62fa3460 2023-07-14T18:10:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59beffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fa2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59beffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fa2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52682/0 (socket says 172.21.15.115:52682) 2023-07-14T18:10:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59beffd700 1 -- 172.21.15.115:0/2472546663 learned_addr learned my addr 172.21.15.115:0/2472546663 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59be7fc700 1 --2- 172.21.15.115:0/2472546663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa2860 0x55bc62fa2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59bf7fe700 1 --2- 172.21.15.115:0/2472546663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa3460 0x55bc62fa38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59beffd700 1 -- 172.21.15.115:0/2472546663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa3460 msgr2=0x55bc62fa38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:10:25.413 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59beffd700 1 --2- 172.21.15.115:0/2472546663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa3460 0x55bc62fa38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59beffd700 1 -- 172.21.15.115:0/2472546663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa2860 msgr2=0x55bc62fa2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:10:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59beffd700 1 --2- 172.21.15.115:0/2472546663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa2860 0x55bc62fa2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59beffd700 1 -- 172.21.15.115:0/2472546663 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc62d20d90 con 0x55bc62fa1f10 2023-07-14T18:10:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59beffd700 1 --2- 172.21.15.115:0/2472546663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fa2320 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f59b00a1950 tx=0x7f59b00995f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495b2bc6dbc2ce5e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59a57fa700 1 -- 172.21.15.115:0/2472546663 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b0009bb0 con 0x55bc62fa1f10 2023-07-14T18:10:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59a57fa700 1 -- 172.21.15.115:0/2472546663 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f59b009ae90 con 0x55bc62fa1f10 2023-07-14T18:10:25.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.367+0000 7f59a57fa700 1 -- 172.21.15.115:0/2472546663 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b0097ef0 con 0x55bc62fa1f10 2023-07-14T18:10:25.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 -- 172.21.15.115:0/2472546663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 msgr2=0x55bc62fa2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 --2- 172.21.15.115:0/2472546663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fa2320 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f59b00a1950 tx=0x7f59b00995f0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 -- 172.21.15.115:0/2472546663 shutdown_connections 2023-07-14T18:10:25.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 --2- 172.21.15.115:0/2472546663 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa3460 0x55bc62fa38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 --2- 172.21.15.115:0/2472546663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fa2320 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 --2- 172.21.15.115:0/2472546663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa2860 0x55bc62fa2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 -- 172.21.15.115:0/2472546663 >> 172.21.15.115:0/2472546663 conn(0x55bc62f97630 msgr2=0x55bc62f9c980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:25.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 -- 172.21.15.115:0/2472546663 shutdown_connections 2023-07-14T18:10:25.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 -- 172.21.15.115:0/2472546663 wait complete. 2023-07-14T18:10:25.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 -- start start 2023-07-14T18:10:25.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 Processor -- start 2023-07-14T18:10:25.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.368+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 0x55bc62fabe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fb2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.416 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59d168eb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc62e92a70 con 0x55bc62fa1f10 2023-07-14T18:10:25.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59d168eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc62e92770 con 0x55bc62fa3460 2023-07-14T18:10:25.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59d168eb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc62d768d0 con 0x55bc62fa2860 2023-07-14T18:10:25.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59d168eb80 10 client.0 _ll_register_callbacks cb 0x55bc62e90710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:10:25.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59beffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59beffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fab900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52688/0 (socket says 172.21.15.115:52688) 2023-07-14T18:10:25.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59beffd700 1 -- 172.21.15.115:0/3064799750 learned_addr learned my addr 172.21.15.115:0/3064799750 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:25.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59be7fc700 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 0x55bc62fabe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59bf7fe700 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fb2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59beffd700 1 -- 172.21.15.115:0/3064799750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 msgr2=0x55bc62fabe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:10:25.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59beffd700 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 0x55bc62fabe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59beffd700 1 -- 172.21.15.115:0/3064799750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 msgr2=0x55bc62fb2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:10:25.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59beffd700 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fb2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59beffd700 1 -- 172.21.15.115:0/3064799750 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c019df00 con 0x55bc62fa1f10 2023-07-14T18:10:25.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59bf7fe700 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fb2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:25.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59beffd700 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fab900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009520 tx=0x7f59b0095700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45493e418208d11a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:25.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59a2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:10:25.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59a2ff5700 1 -- 172.21.15.115:0/3064799750 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b0009bb0 con 0x55bc62fa1f10 2023-07-14T18:10:25.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59a2ff5700 1 -- 172.21.15.115:0/3064799750 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f59b009da60 con 0x55bc62fa1f10 2023-07-14T18:10:25.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59d168eb80 1 -- 172.21.15.115:0/3064799750 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc62e4c4a0 con 0x55bc62fa1f10 2023-07-14T18:10:25.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59a2ff5700 1 -- 172.21.15.115:0/3064799750 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59b00a19f0 con 0x55bc62fa1f10 2023-07-14T18:10:25.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59d168eb80 10 client.14757 fetch_fsmap learned FSMap version 17 2023-07-14T18:10:25.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.369+0000 7f59d168eb80 1 -- 172.21.15.115:0/3064799750 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55bc62d09c10 con 0x55bc62fa1f10 2023-07-14T18:10:25.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59a2ff5700 1 -- 172.21.15.115:0/3064799750 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55bc62e4c4a0 con 0x55bc62fa1f10 2023-07-14T18:10:25.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59a2ff5700 1 -- 172.21.15.115:0/3064799750 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f59b00c6060 con 0x55bc62fa1f10 2023-07-14T18:10:25.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59d168eb80 10 client.14757 fetch_fsmap finished waiting for FSMap version 17 2023-07-14T18:10:25.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59d168eb80 10 client.14757 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:10:25.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59d168eb80 20 client.14757 populate_metadata read hostname 'smithi115' 2023-07-14T18:10:25.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59d168eb80 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x55bc62fb6d00 0x55bc62fb5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59d168eb80 4 client.14757 mds_command: new command op to 24515 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:10:25.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59d168eb80 1 -- 172.21.15.115:0/3064799750 --> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc62f70d50 con 0x55bc62fb6d00 2023-07-14T18:10:25.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59be7fc700 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x55bc62fb6d00 0x55bc62fb5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59a2ff5700 10 client.14757 ms_handle_connect on v2:172.21.15.150:6838/2895627668 2023-07-14T18:10:25.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.370+0000 7f59be7fc700 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x55bc62fb6d00 0x55bc62fb5db0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be4ee636629f3fa7 server_cookie=3c6bf64e36401f78 in_seq=0 out_seq=0 2023-07-14T18:10:25.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59a2ff5700 1 -- 172.21.15.115:0/3064799750 <== mds.0 v2:172.21.15.150:6838/2895627668 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55bc62f70d50 con 0x55bc62fb6d00 2023-07-14T18:10:25.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59a2ff5700 10 client.14757 handle_command_reply: tid=0 2023-07-14T18:10:25.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 client.14757 shutdown 2023-07-14T18:10:25.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 -- 172.21.15.115:0/3064799750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 msgr2=0x55bc62fab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fab900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009520 tx=0x7f59b0095700 comp rx=0 tx=0).stop 2023-07-14T18:10:25.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 -- 172.21.15.115:0/3064799750 shutdown_connections 2023-07-14T18:10:25.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x55bc62fb6d00 0x55bc62fb5db0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fb2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59a2ff5700 0 client.14757 ms_handle_reset on v2:172.21.15.150:6838/2895627668 2023-07-14T18:10:25.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62fab900 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 --2- 172.21.15.115:0/3064799750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 0x55bc62fabe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 -- 172.21.15.115:0/3064799750 >> 172.21.15.115:0/3064799750 conn(0x55bc62f97630 msgr2=0x55bc62f9b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:25.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.371+0000 7f59d168eb80 1 -- 172.21.15.115:0/3064799750 shutdown_connections 2023-07-14T18:10:25.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.372+0000 7f59d168eb80 1 -- 172.21.15.115:0/3064799750 wait complete. 2023-07-14T18:10:25.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.372+0000 7f59d168eb80 20 client.14757 trim_cache size 0 max 16384 2023-07-14T18:10:25.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59d168eb80 1 Processor -- start 2023-07-14T18:10:25.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59d168eb80 1 -- start start 2023-07-14T18:10:25.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62f9c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 0x55bc62fa6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fa6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59d168eb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc62f9dc80 con 0x55bc62fa1f10 2023-07-14T18:10:25.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59d168eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc62e9c470 con 0x55bc62fa3460 2023-07-14T18:10:25.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59d168eb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc62ec1790 con 0x55bc62fa2860 2023-07-14T18:10:25.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59beffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62f9c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59beffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62f9c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52696/0 (socket says 172.21.15.115:52696) 2023-07-14T18:10:25.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59beffd700 1 -- 172.21.15.115:0/2069041233 learned_addr learned my addr 172.21.15.115:0/2069041233 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:25.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59be7fc700 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 0x55bc62fa6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59bf7fe700 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fa6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59beffd700 1 -- 172.21.15.115:0/2069041233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 msgr2=0x55bc62fa6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:10:25.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59beffd700 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 0x55bc62fa6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59beffd700 1 -- 172.21.15.115:0/2069041233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 msgr2=0x55bc62fa6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:10:25.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59beffd700 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fa6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59beffd700 1 -- 172.21.15.115:0/2069041233 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc62d47200 con 0x55bc62fa1f10 2023-07-14T18:10:25.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59bf7fe700 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fa6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:25.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59beffd700 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62f9c0a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f59b0097d80 tx=0x7f59b0095770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fedb35afd86943 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:25.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59a5ffb700 1 -- 172.21.15.115:0/2069041233 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b00a1ef0 con 0x55bc62fa1f10 2023-07-14T18:10:25.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59a5ffb700 1 -- 172.21.15.115:0/2069041233 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f59b0009bb0 con 0x55bc62fa1f10 2023-07-14T18:10:25.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.389+0000 7f59a5ffb700 1 -- 172.21.15.115:0/2069041233 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b0084300 con 0x55bc62fa1f10 2023-07-14T18:10:25.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 -- 172.21.15.115:0/2069041233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 msgr2=0x55bc62f9c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62f9c0a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f59b0097d80 tx=0x7f59b0095770 comp rx=0 tx=0).stop 2023-07-14T18:10:25.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 -- 172.21.15.115:0/2069041233 shutdown_connections 2023-07-14T18:10:25.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc62fa6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa1f10 0x55bc62f9c0a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 --2- 172.21.15.115:0/2069041233 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa2860 0x55bc62fa6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 -- 172.21.15.115:0/2069041233 >> 172.21.15.115:0/2069041233 conn(0x55bc62f97630 msgr2=0x55bc62fb5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:25.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 -- 172.21.15.115:0/2069041233 shutdown_connections 2023-07-14T18:10:25.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 -- 172.21.15.115:0/2069041233 wait complete. 2023-07-14T18:10:25.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 -- start start 2023-07-14T18:10:25.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.390+0000 7f59d168eb80 1 Processor -- start 2023-07-14T18:10:25.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa1f10 0x55bc6300a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa2860 0x55bc6300c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc6300edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6300f460 con 0x55bc62fa2860 2023-07-14T18:10:25.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6300f5d0 con 0x55bc62fa3460 2023-07-14T18:10:25.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6300f790 con 0x55bc62fa1f10 2023-07-14T18:10:25.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 10 client.0 _ll_register_callbacks cb 0x55bc62e90710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:10:25.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59be7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa2860 0x55bc6300c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59be7fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa2860 0x55bc6300c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52704/0 (socket says 172.21.15.115:52704) 2023-07-14T18:10:25.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59be7fc700 1 -- 172.21.15.115:0/395384539 learned_addr learned my addr 172.21.15.115:0/395384539 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:25.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59beffd700 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa1f10 0x55bc6300a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59bf7fe700 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc6300edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59be7fc700 1 -- 172.21.15.115:0/395384539 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa1f10 msgr2=0x55bc6300a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:10:25.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59be7fc700 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa1f10 0x55bc6300a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59be7fc700 1 -- 172.21.15.115:0/395384539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 msgr2=0x55bc6300edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:10:25.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59be7fc700 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc6300edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59be7fc700 1 -- 172.21.15.115:0/395384539 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc62d09c10 con 0x55bc62fa2860 2023-07-14T18:10:25.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59be7fc700 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa2860 0x55bc6300c8a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009990 tx=0x7f59b4009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa578c408a164810 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:25.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59a2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:10:25.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59a2ff5700 1 -- 172.21.15.115:0/395384539 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b4005420 con 0x55bc62fa2860 2023-07-14T18:10:25.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59a2ff5700 1 -- 172.21.15.115:0/395384539 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f59b4005580 con 0x55bc62fa2860 2023-07-14T18:10:25.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 1 -- 172.21.15.115:0/395384539 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59b009ae90 con 0x55bc62fa2860 2023-07-14T18:10:25.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59a2ff5700 1 -- 172.21.15.115:0/395384539 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59b4032070 con 0x55bc62fa2860 2023-07-14T18:10:25.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 10 client.14763 fetch_fsmap learned FSMap version 17 2023-07-14T18:10:25.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.391+0000 7f59d168eb80 1 -- 172.21.15.115:0/395384539 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55bc62d1f260 con 0x55bc62fa2860 2023-07-14T18:10:25.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59a2ff5700 1 -- 172.21.15.115:0/395384539 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f59b009ae90 con 0x55bc62fa2860 2023-07-14T18:10:25.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59a2ff5700 1 -- 172.21.15.115:0/395384539 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f59b4004500 con 0x55bc62fa2860 2023-07-14T18:10:25.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59d168eb80 10 client.14763 fetch_fsmap finished waiting for FSMap version 17 2023-07-14T18:10:25.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59d168eb80 10 client.14763 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:10:25.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59d168eb80 20 client.14763 populate_metadata read hostname 'smithi115' 2023-07-14T18:10:25.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59d168eb80 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x55bc630144d0 0x55bc63010e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:25.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59d168eb80 4 client.14763 mds_command: new command op to 24515 tid=0[{"prefix": "status"}] 2023-07-14T18:10:25.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59d168eb80 1 -- 172.21.15.115:0/395384539 --> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bc62eebe60 con 0x55bc630144d0 2023-07-14T18:10:25.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59beffd700 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x55bc630144d0 0x55bc63010e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:25.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59a2ff5700 10 client.14763 ms_handle_connect on v2:172.21.15.150:6838/2895627668 2023-07-14T18:10:25.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.392+0000 7f59beffd700 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x55bc630144d0 0x55bc63010e40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f9882202caca0f5 server_cookie=cd1d1b3eaf5e3e73 in_seq=0 out_seq=0 2023-07-14T18:10:25.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59a2ff5700 1 -- 172.21.15.115:0/395384539 <== mds.0 v2:172.21.15.150:6838/2895627668 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55bc62eebe60 con 0x55bc630144d0 2023-07-14T18:10:25.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59a2ff5700 10 client.14763 handle_command_reply: tid=0 2023-07-14T18:10:25.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 client.14763 shutdown 2023-07-14T18:10:25.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 -- 172.21.15.115:0/395384539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa2860 msgr2=0x55bc6300c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa2860 0x55bc6300c8a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009990 tx=0x7f59b4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 -- 172.21.15.115:0/395384539 shutdown_connections 2023-07-14T18:10:25.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x55bc630144d0 0x55bc63010e40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc62fa3460 0x55bc6300edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59a2ff5700 0 client.14763 ms_handle_reset on v2:172.21.15.150:6838/2895627668 2023-07-14T18:10:25.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc62fa1f10 0x55bc6300a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 --2- 172.21.15.115:0/395384539 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc62fa2860 0x55bc6300c8a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 -- 172.21.15.115:0/395384539 >> 172.21.15.115:0/395384539 conn(0x55bc62f97630 msgr2=0x55bc62f9c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:25.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 -- 172.21.15.115:0/395384539 shutdown_connections 2023-07-14T18:10:25.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 1 -- 172.21.15.115:0/395384539 wait complete. 2023-07-14T18:10:25.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59d168eb80 20 client.14763 trim_cache size 0 max 16384 2023-07-14T18:10:25.443 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:10:25.443 INFO:teuthology.orchestra.run.smithi115.stdout: "cluster_fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:10:25.443 INFO:teuthology.orchestra.run.smithi115.stdout: "whoami": 0, 2023-07-14T18:10:25.443 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 24515, 2023-07-14T18:10:25.444 INFO:teuthology.orchestra.run.smithi115.stdout: "want_state": "up:active", 2023-07-14T18:10:25.444 INFO:teuthology.orchestra.run.smithi115.stdout: "state": "up:active", 2023-07-14T18:10:25.444 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs", 2023-07-14T18:10:25.444 INFO:teuthology.orchestra.run.smithi115.stdout: "rank_uptime": 6.98279856, 2023-07-14T18:10:25.444 INFO:teuthology.orchestra.run.smithi115.stdout: "mdsmap_epoch": 17, 2023-07-14T18:10:25.444 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch": 87, 2023-07-14T18:10:25.444 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T18:10:25.445 INFO:teuthology.orchestra.run.smithi115.stdout: "uptime": 13.834757722999999 2023-07-14T18:10:25.445 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:10:25.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 -- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59a80a1150 msgr2=0x7f59a80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59a80a1150 0x7f59a80a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f59b4000c00 tx=0x7f59b4005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 -- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00feb70 msgr2=0x7f59c01958c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:25.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00feb70 0x7f59c01958c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59b000d980 tx=0x7f59b000dd40 comp rx=0 tx=0).stop 2023-07-14T18:10:25.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 -- 172.21.15.115:0/4113773384 shutdown_connections 2023-07-14T18:10:25.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f59a80a1150 0x7f59a80a3600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59c01006f0 0x7f59c019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f59c00feb70 0x7f59c01958c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 --2- 172.21.15.115:0/4113773384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c00ffd60 0x7f59c0195e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:25.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 -- 172.21.15.115:0/4113773384 >> 172.21.15.115:0/4113773384 conn(0x7f59c00fa1d0 msgr2=0x7f59c00fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:25.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 -- 172.21.15.115:0/4113773384 shutdown_connections 2023-07-14T18:10:25.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:25.393+0000 7f59c54f6700 1 -- 172.21.15.115:0/4113773384 wait complete. 2023-07-14T18:10:25.448 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T18:10:25.449 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:10:25.449 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T18:10:25.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:25 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/512418092' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:25.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:25 smithi115 ceph-mon[127451]: pgmap v255: 129 pgs: 105 active+clean, 15 creating+peering, 9 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 9 op/s 2023-07-14T18:10:25.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:25 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3914532483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:25.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:25 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/991634724' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:25.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:25 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/512418092' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:25.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:25 smithi150 ceph-mon[128853]: pgmap v255: 129 pgs: 105 active+clean, 15 creating+peering, 9 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 9 op/s 2023-07-14T18:10:25.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:25 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3914532483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:25.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:25 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/991634724' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:25.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:25 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/512418092' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:25.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:25 smithi187 ceph-mon[129317]: pgmap v255: 129 pgs: 105 active+clean, 15 creating+peering, 9 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 9 op/s 2023-07-14T18:10:25.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:25 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3914532483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:25.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:25 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/991634724' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:25.993 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:27.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.297+0000 7f626b814700 1 -- 172.21.15.115:0/2758588445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 msgr2=0x7f626410cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:27.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.297+0000 7f626b814700 1 --2- 172.21.15.115:0/2758588445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 0x7f626410cb50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f625c00b0d0 tx=0x7f625c00b3e0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 -- 172.21.15.115:0/2758588445 shutdown_connections 2023-07-14T18:10:27.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 --2- 172.21.15.115:0/2758588445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 0x7f626410cb50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 --2- 172.21.15.115:0/2758588445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640fff30 0x7f6264100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 --2- 172.21.15.115:0/2758588445 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62641008d0 0x7f62641048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 -- 172.21.15.115:0/2758588445 >> 172.21.15.115:0/2758588445 conn(0x7f62640fb790 msgr2=0x7f62640fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:27.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 -- 172.21.15.115:0/2758588445 shutdown_connections 2023-07-14T18:10:27.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 -- 172.21.15.115:0/2758588445 wait complete. 2023-07-14T18:10:27.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 Processor -- start 2023-07-14T18:10:27.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 -- start start 2023-07-14T18:10:27.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640fff30 0x7f6264072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:27.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62641008d0 0x7f6264072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:27.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 0x7f6264073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:27.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641a4160 con 0x7f6264104fd0 2023-07-14T18:10:27.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641a42d0 con 0x7f62640fff30 2023-07-14T18:10:27.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.298+0000 7f626b814700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641a4490 con 0x7f62641008d0 2023-07-14T18:10:27.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6269db1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 0x7f6264073400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:27.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6269db1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 0x7f6264073400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52716/0 (socket says 172.21.15.115:52716) 2023-07-14T18:10:27.340 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6269db1700 1 -- 172.21.15.115:0/2741284473 learned_addr learned my addr 172.21.15.115:0/2741284473 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:27.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6268daf700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62641008d0 0x7f6264072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:27.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f62695b0700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640fff30 0x7f6264072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:27.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6269db1700 1 -- 172.21.15.115:0/2741284473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62641008d0 msgr2=0x7f6264072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:27.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6269db1700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62641008d0 0x7f6264072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6269db1700 1 -- 172.21.15.115:0/2741284473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640fff30 msgr2=0x7f6264072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:27.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6269db1700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640fff30 0x7f6264072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6269db1700 1 -- 172.21.15.115:0/2741284473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c009d00 con 0x7f6264104fd0 2023-07-14T18:10:27.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f62695b0700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640fff30 0x7f6264072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:27.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f6269db1700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 0x7f6264073400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f625c009380 tx=0x7f625c0093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:27.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f625a7fc700 1 -- 172.21.15.115:0/2741284473 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c00e040 con 0x7f6264104fd0 2023-07-14T18:10:27.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f625a7fc700 1 -- 172.21.15.115:0/2741284473 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f625c003e10 con 0x7f6264104fd0 2023-07-14T18:10:27.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62641a4710 con 0x7f6264104fd0 2023-07-14T18:10:27.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f625a7fc700 1 -- 172.21.15.115:0/2741284473 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c01ba00 con 0x7f6264104fd0 2023-07-14T18:10:27.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.299+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62641a4b20 con 0x7f6264104fd0 2023-07-14T18:10:27.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.300+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626404e340 con 0x7f6264104fd0 2023-07-14T18:10:27.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.301+0000 7f625a7fc700 1 -- 172.21.15.115:0/2741284473 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f625c024410 con 0x7f6264104fd0 2023-07-14T18:10:27.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.302+0000 7f625a7fc700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f62500a1150 0x7f62500a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:27.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.302+0000 7f625a7fc700 1 -- 172.21.15.115:0/2741284473 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f625c0ce1f0 con 0x7f6264104fd0 2023-07-14T18:10:27.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.302+0000 7f62695b0700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f62500a1150 0x7f62500a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:27.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.302+0000 7f62695b0700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f62500a1150 0x7f62500a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6260005d90 tx=0x7f6260005d00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:27.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.306+0000 7f625a7fc700 1 -- 172.21.15.115:0/2741284473 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f625c097890 con 0x7f6264104fd0 2023-07-14T18:10:27.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.533+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f62641a4ea0 con 0x7f6264104fd0 2023-07-14T18:10:27.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.533+0000 7f625a7fc700 1 -- 172.21.15.115:0/2741284473 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f625c09aeb0 con 0x7f6264104fd0 2023-07-14T18:10:27.572 INFO:teuthology.orchestra.run.smithi115.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-14T18:10:27.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f62500a1150 msgr2=0x7f62500a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:27.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f62500a1150 0x7f62500a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6260005d90 tx=0x7f6260005d00 comp rx=0 tx=0).stop 2023-07-14T18:10:27.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 msgr2=0x7f6264073400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:27.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 0x7f6264073400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f625c009380 tx=0x7f625c0093b0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 shutdown_connections 2023-07-14T18:10:27.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f62500a1150 0x7f62500a3600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6264104fd0 0x7f6264073400 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640fff30 0x7f6264072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 --2- 172.21.15.115:0/2741284473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f62641008d0 0x7f6264072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:27.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 >> 172.21.15.115:0/2741284473 conn(0x7f62640fb790 msgr2=0x7f62640fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:27.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.535+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 shutdown_connections 2023-07-14T18:10:27.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:27.536+0000 7f626b814700 1 -- 172.21.15.115:0/2741284473 wait complete. 2023-07-14T18:10:28.037 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-07-14T18:10:28.038 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T18:10:28.038 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi115.front.sepia.ceph.com 2023-07-14T18:10:28.038 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-07-14T18:10:28.038 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:10:28.039 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-07-14T18:10:28.039 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T18:10:28.039 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T18:10:28.039 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-07-14T18:10:28.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:28.040 DEBUG:teuthology.orchestra.run.smithi115:> ip addr 2023-07-14T18:10:28.059 INFO:teuthology.orchestra.run.smithi115.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T18:10:28.059 INFO:teuthology.orchestra.run.smithi115.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T18:10:28.059 INFO:teuthology.orchestra.run.smithi115.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T18:10:28.059 INFO:teuthology.orchestra.run.smithi115.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:10:28.060 INFO:teuthology.orchestra.run.smithi115.stdout: inet6 ::1/128 scope host 2023-07-14T18:10:28.060 INFO:teuthology.orchestra.run.smithi115.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:10:28.060 INFO:teuthology.orchestra.run.smithi115.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:10:28.060 INFO:teuthology.orchestra.run.smithi115.stdout: link/ether 0c:c4:7a:d8:a5:44 brd ff:ff:ff:ff:ff:ff 2023-07-14T18:10:28.060 INFO:teuthology.orchestra.run.smithi115.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:10:28.060 INFO:teuthology.orchestra.run.smithi115.stdout: link/ether 0c:c4:7a:d8:a5:45 brd ff:ff:ff:ff:ff:ff 2023-07-14T18:10:28.060 INFO:teuthology.orchestra.run.smithi115.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:10:28.061 INFO:teuthology.orchestra.run.smithi115.stdout: link/ether 0c:c4:7a:8f:ce:8e brd ff:ff:ff:ff:ff:ff 2023-07-14T18:10:28.061 INFO:teuthology.orchestra.run.smithi115.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T18:10:28.061 INFO:teuthology.orchestra.run.smithi115.stdout: link/ether 0c:c4:7a:8f:ce:8f brd ff:ff:ff:ff:ff:ff 2023-07-14T18:10:28.061 INFO:teuthology.orchestra.run.smithi115.stdout: inet 172.21.15.115/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-14T18:10:28.061 INFO:teuthology.orchestra.run.smithi115.stdout: valid_lft 41482sec preferred_lft 41482sec 2023-07-14T18:10:28.061 INFO:teuthology.orchestra.run.smithi115.stdout: inet6 fe80::1d33:d16a:ed86:b5bf/64 scope link noprefixroute 2023-07-14T18:10:28.061 INFO:teuthology.orchestra.run.smithi115.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:10:28.062 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T18:10:28.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:28.063 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:10:28.063 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:10:28.063 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link add name ceph-brx type bridge 2023-07-14T18:10:28.063 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip addr flush dev ceph-brx 2023-07-14T18:10:28.064 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link set ceph-brx up 2023-07-14T18:10:28.064 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T18:10:28.064 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:10:28.223 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:27 smithi115 ceph-mon[127451]: pgmap v256: 129 pgs: 116 active+clean, 13 creating+peering; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2023-07-14T18:10:28.224 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:27 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2741284473' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:10:28.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:28.226 DEBUG:teuthology.orchestra.run.smithi115:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T18:10:28.256 INFO:teuthology.orchestra.run.smithi115.stdout:1 2023-07-14T18:10:28.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:28.258 DEBUG:teuthology.orchestra.run.smithi115:> route 2023-07-14T18:10:28.333 INFO:teuthology.orchestra.run.smithi115.stdout:Kernel IP routing table 2023-07-14T18:10:28.333 INFO:teuthology.orchestra.run.smithi115.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T18:10:28.333 INFO:teuthology.orchestra.run.smithi115.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-14T18:10:28.333 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-14T18:10:28.333 INFO:teuthology.orchestra.run.smithi115.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T18:10:28.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:28.334 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:10:28.335 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:10:28.335 DEBUG:teuthology.orchestra.run.smithi115:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-14T18:10:28.335 DEBUG:teuthology.orchestra.run.smithi115:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-14T18:10:28.335 DEBUG:teuthology.orchestra.run.smithi115:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-14T18:10:28.336 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:10:28.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:27 smithi150 ceph-mon[128853]: pgmap v256: 129 pgs: 116 active+clean, 13 creating+peering; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2023-07-14T18:10:28.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:27 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2741284473' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:10:28.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:27 smithi187 ceph-mon[129317]: pgmap v256: 129 pgs: 116 active+clean, 13 creating+peering; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 6 op/s 2023-07-14T18:10:28.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:27 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2741284473' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:10:28.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:28.655 DEBUG:teuthology.orchestra.run.smithi115:> ip netns list 2023-07-14T18:10:28.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:28.672 DEBUG:teuthology.orchestra.run.smithi115:> ip netns list-id 2023-07-14T18:10:28.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:28.729 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:10:28.729 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:10:28.729 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-14T18:10:28.730 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-07-14T18:10:28.730 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:10:28.826 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-07-14T18:10:28.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:28.828 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:10:28.828 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:10:28.829 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-07-14T18:10:28.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T18:10:28.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-07-14T18:10:28.831 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-07-14T18:10:28.832 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-07-14T18:10:28.832 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:10:29.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:29.019 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:10:29.020 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:10:29.020 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link set brx.0 up 2023-07-14T18:10:29.020 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T18:10:29.020 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:10:29.120 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-07-14T18:10:29.120 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T18:10:29.121 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:10:29.137 INFO:teuthology.orchestra.run.smithi115.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-07-14T18:10:29.137 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T18:10:29.138 DEBUG:teuthology.orchestra.run.smithi115:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:10:29.193 DEBUG:teuthology.orchestra.run.smithi115:> sudo modprobe fuse 2023-07-14T18:10:29.261 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T18:10:29.262 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T18:10:29.328 INFO:teuthology.orchestra.run.smithi115.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T18:10:29.329 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T18:10:29.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:29.330 DEBUG:teuthology.orchestra.run.smithi115:> ls /sys/fs/fuse/connections 2023-07-14T18:10:29.386 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-14T18:10:29.387 DEBUG:teuthology.orchestra.run.smithi115:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-07-14T18:10:29.429 DEBUG:teuthology.orchestra.run.smithi115:> sudo modprobe fuse 2023-07-14T18:10:29.455 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T18:10:29.456 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T18:10:29.522 INFO:teuthology.orchestra.run.smithi115.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T18:10:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T18:10:29.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:29.525 DEBUG:teuthology.orchestra.run.smithi115:> ls /sys/fs/fuse/connections 2023-07-14T18:10:29.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi115.stderr:2023-07-14T18:10:29.517+0000 7f3a32f855c0 -1 init, newargv = 0x55a59c9eace0 newargc=15 2023-07-14T18:10:29.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi115.stderr:ceph-fuse[164740]: starting ceph client 2023-07-14T18:10:29.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi115.stderr:ceph-fuse[164740]: starting fuse 2023-07-14T18:10:29.574 INFO:teuthology.orchestra.run.smithi115.stdout:124 2023-07-14T18:10:29.574 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-07-14T18:10:29.575 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-14T18:10:29.575 DEBUG:teuthology.orchestra.run.smithi115:> import glob 2023-07-14T18:10:29.575 DEBUG:teuthology.orchestra.run.smithi115:> import re 2023-07-14T18:10:29.576 DEBUG:teuthology.orchestra.run.smithi115:> import os 2023-07-14T18:10:29.576 DEBUG:teuthology.orchestra.run.smithi115:> import subprocess 2023-07-14T18:10:29.576 DEBUG:teuthology.orchestra.run.smithi115:> 2023-07-14T18:10:29.576 DEBUG:teuthology.orchestra.run.smithi115:> def _find_admin_socket(client_name): 2023-07-14T18:10:29.577 DEBUG:teuthology.orchestra.run.smithi115:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-07-14T18:10:29.577 DEBUG:teuthology.orchestra.run.smithi115:> files = glob.glob(asok_path) 2023-07-14T18:10:29.577 DEBUG:teuthology.orchestra.run.smithi115:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-07-14T18:10:29.577 DEBUG:teuthology.orchestra.run.smithi115:> 2023-07-14T18:10:29.578 DEBUG:teuthology.orchestra.run.smithi115:> # Given a non-glob path, it better be there 2023-07-14T18:10:29.578 DEBUG:teuthology.orchestra.run.smithi115:> if "*" not in asok_path: 2023-07-14T18:10:29.578 DEBUG:teuthology.orchestra.run.smithi115:> assert(len(files) == 1) 2023-07-14T18:10:29.579 DEBUG:teuthology.orchestra.run.smithi115:> return files[0] 2023-07-14T18:10:29.579 DEBUG:teuthology.orchestra.run.smithi115:> 2023-07-14T18:10:29.579 DEBUG:teuthology.orchestra.run.smithi115:> for f in files: 2023-07-14T18:10:29.579 DEBUG:teuthology.orchestra.run.smithi115:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-14T18:10:29.580 DEBUG:teuthology.orchestra.run.smithi115:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-14T18:10:29.580 DEBUG:teuthology.orchestra.run.smithi115:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-14T18:10:29.580 DEBUG:teuthology.orchestra.run.smithi115:> contents = proc_f.read() 2023-07-14T18:10:29.580 DEBUG:teuthology.orchestra.run.smithi115:> if mountpoint in contents: 2023-07-14T18:10:29.581 DEBUG:teuthology.orchestra.run.smithi115:> return f 2023-07-14T18:10:29.581 DEBUG:teuthology.orchestra.run.smithi115:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-14T18:10:29.581 DEBUG:teuthology.orchestra.run.smithi115:> 2023-07-14T18:10:29.582 DEBUG:teuthology.orchestra.run.smithi115:> print(_find_admin_socket("client.admin")) 2023-07-14T18:10:29.582 DEBUG:teuthology.orchestra.run.smithi115:> ' 2023-07-14T18:10:29.701 INFO:teuthology.orchestra.run.smithi115.stdout:/var/run/ceph/ceph-client.admin.164740.asok 2023-07-14T18:10:29.881 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.164740.asok 2023-07-14T18:10:29.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:29.882 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.164740.asok status 2023-07-14T18:10:30.045 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:10:30.045 INFO:teuthology.orchestra.run.smithi115.stdout: "metadata": { 2023-07-14T18:10:30.045 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_sha1": "81c9387d2e7576da065747ec9c16d1c7d69016ec", 2023-07-14T18:10:30.045 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:10:30.045 INFO:teuthology.orchestra.run.smithi115.stdout: "entity_id": "admin", 2023-07-14T18:10:30.046 INFO:teuthology.orchestra.run.smithi115.stdout: "hostname": "smithi115", 2023-07-14T18:10:30.046 INFO:teuthology.orchestra.run.smithi115.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-07-14T18:10:30.046 INFO:teuthology.orchestra.run.smithi115.stdout: "pid": "164740", 2023-07-14T18:10:30.046 INFO:teuthology.orchestra.run.smithi115.stdout: "root": "/" 2023-07-14T18:10:30.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-07-14T18:10:30.046 INFO:teuthology.orchestra.run.smithi115.stdout: "dentry_count": 0, 2023-07-14T18:10:30.046 INFO:teuthology.orchestra.run.smithi115.stdout: "dentry_pinned_count": 0, 2023-07-14T18:10:30.047 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 14775, 2023-07-14T18:10:30.047 INFO:teuthology.orchestra.run.smithi115.stdout: "inst": { 2023-07-14T18:10:30.047 INFO:teuthology.orchestra.run.smithi115.stdout: "name": { 2023-07-14T18:10:30.047 INFO:teuthology.orchestra.run.smithi115.stdout: "type": "client", 2023-07-14T18:10:30.047 INFO:teuthology.orchestra.run.smithi115.stdout: "num": 14775 2023-07-14T18:10:30.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-07-14T18:10:30.047 INFO:teuthology.orchestra.run.smithi115.stdout: "addr": { 2023-07-14T18:10:30.047 INFO:teuthology.orchestra.run.smithi115.stdout: "type": "v1", 2023-07-14T18:10:30.048 INFO:teuthology.orchestra.run.smithi115.stdout: "addr": "192.168.0.1:0", 2023-07-14T18:10:30.048 INFO:teuthology.orchestra.run.smithi115.stdout: "nonce": 2687031739 2023-07-14T18:10:30.048 INFO:teuthology.orchestra.run.smithi115.stdout: } 2023-07-14T18:10:30.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-07-14T18:10:30.048 INFO:teuthology.orchestra.run.smithi115.stdout: "addr": { 2023-07-14T18:10:30.048 INFO:teuthology.orchestra.run.smithi115.stdout: "type": "v1", 2023-07-14T18:10:30.048 INFO:teuthology.orchestra.run.smithi115.stdout: "addr": "192.168.0.1:0", 2023-07-14T18:10:30.048 INFO:teuthology.orchestra.run.smithi115.stdout: "nonce": 2687031739 2023-07-14T18:10:30.049 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-07-14T18:10:30.049 INFO:teuthology.orchestra.run.smithi115.stdout: "inst_str": "client.14775 192.168.0.1:0/2687031739", 2023-07-14T18:10:30.049 INFO:teuthology.orchestra.run.smithi115.stdout: "addr_str": "192.168.0.1:0/2687031739", 2023-07-14T18:10:30.049 INFO:teuthology.orchestra.run.smithi115.stdout: "inode_count": 1, 2023-07-14T18:10:30.049 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_epoch": 17, 2023-07-14T18:10:30.049 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_epoch": 87, 2023-07-14T18:10:30.049 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_epoch_barrier": 0, 2023-07-14T18:10:30.049 INFO:teuthology.orchestra.run.smithi115.stdout: "blocklisted": false, 2023-07-14T18:10:30.050 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs" 2023-07-14T18:10:30.050 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:10:30.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:30.056 DEBUG:teuthology.orchestra.run.smithi115:> stat --file-system '--printf=%T 2023-07-14T18:10:30.056 DEBUG:teuthology.orchestra.run.smithi115:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:10:30.073 INFO:teuthology.orchestra.run.smithi115.stdout:fuseblk 2023-07-14T18:10:30.073 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:10:30.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:30.074 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:10:30.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:30.140 DEBUG:teuthology.orchestra.run.smithi115:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-07-14T18:10:30.244 INFO:teuthology.orchestra.run.smithi115.stdout:# file: . 2023-07-14T18:10:30.244 INFO:teuthology.orchestra.run.smithi115.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-07-14T18:10:30.245 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:30.245 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:10:30.245 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:10:30.263 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:10:30.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:29 smithi115 ceph-mon[127451]: pgmap v257: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 412 B/s wr, 3 op/s 2023-07-14T18:10:30.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:29 smithi150 ceph-mon[128853]: pgmap v257: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 412 B/s wr, 3 op/s 2023-07-14T18:10:30.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 -- 172.21.15.115:0/2340501349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d680feb00 msgr2=0x7f9d680fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:30.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/2340501349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d680feb00 0x7f9d680fef10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9d58009990 tx=0x7f9d58009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 -- 172.21.15.115:0/2340501349 shutdown_connections 2023-07-14T18:10:30.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/2340501349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d68100680 0x7f9d68106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/2340501349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d680feb00 0x7f9d680fef10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/2340501349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d680ffcf0 0x7f9d68100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 -- 172.21.15.115:0/2340501349 >> 172.21.15.115:0/2340501349 conn(0x7f9d680fa100 msgr2=0x7f9d680fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:30.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 -- 172.21.15.115:0/2340501349 shutdown_connections 2023-07-14T18:10:30.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 -- 172.21.15.115:0/2340501349 wait complete. 2023-07-14T18:10:30.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 Processor -- start 2023-07-14T18:10:30.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.388+0000 7f9d6e26a700 1 -- start start 2023-07-14T18:10:30.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6e26a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d680feb00 0x7f9d68068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:30.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6e26a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680ffcf0 0x7f9d6819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:30.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6e26a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d68100680 0x7f9d6819ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6e26a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6819d450 con 0x7f9d68100680 2023-07-14T18:10:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6e26a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6819d5c0 con 0x7f9d680ffcf0 2023-07-14T18:10:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6e26a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6819d780 con 0x7f9d680feb00 2023-07-14T18:10:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6c807700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d68100680 0x7f9d6819ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6c807700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d68100680 0x7f9d6819ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52740/0 (socket says 172.21.15.115:52740) 2023-07-14T18:10:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6c807700 1 -- 172.21.15.115:0/758065353 learned_addr learned my addr 172.21.15.115:0/758065353 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d67fff700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d680feb00 0x7f9d68068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d677fe700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680ffcf0 0x7f9d6819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:30.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6c807700 1 -- 172.21.15.115:0/758065353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d680feb00 msgr2=0x7f9d68068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:30.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6c807700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d680feb00 0x7f9d68068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6c807700 1 -- 172.21.15.115:0/758065353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680ffcf0 msgr2=0x7f9d6819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:30.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6c807700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680ffcf0 0x7f9d6819a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6c807700 1 -- 172.21.15.115:0/758065353 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d580095f0 con 0x7f9d68100680 2023-07-14T18:10:30.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6c807700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d68100680 0x7f9d6819ccb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d60007ee0 tx=0x7f9d6000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:30.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d657fa700 1 -- 172.21.15.115:0/758065353 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60017070 con 0x7f9d68100680 2023-07-14T18:10:30.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6819da60 con 0x7f9d68100680 2023-07-14T18:10:30.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d657fa700 1 -- 172.21.15.115:0/758065353 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9d6000f040 con 0x7f9d68100680 2023-07-14T18:10:30.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d657fa700 1 -- 172.21.15.115:0/758065353 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d60005710 con 0x7f9d68100680 2023-07-14T18:10:30.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.389+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6819df70 con 0x7f9d68100680 2023-07-14T18:10:30.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.390+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d681060e0 con 0x7f9d68100680 2023-07-14T18:10:30.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.391+0000 7f9d657fa700 1 -- 172.21.15.115:0/758065353 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f9d60005870 con 0x7f9d68100680 2023-07-14T18:10:30.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.392+0000 7f9d657fa700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9d500a11a0 0x7f9d500a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:30.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.392+0000 7f9d657fa700 1 -- 172.21.15.115:0/758065353 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9d600cc2f0 con 0x7f9d68100680 2023-07-14T18:10:30.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.392+0000 7f9d67fff700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9d500a11a0 0x7f9d500a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:30.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.392+0000 7f9d67fff700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9d500a11a0 0x7f9d500a3650 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c010 tx=0x7f9d58005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:30.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:29 smithi187 ceph-mon[129317]: pgmap v257: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 412 B/s wr, 3 op/s 2023-07-14T18:10:30.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.396+0000 7f9d657fa700 1 -- 172.21.15.115:0/758065353 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9d60095850 con 0x7f9d68100680 2023-07-14T18:10:30.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.598+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d6802c8e0 con 0x7f9d68100680 2023-07-14T18:10:30.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.600+0000 7f9d657fa700 1 -- 172.21.15.115:0/758065353 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7f9d60098e70 con 0x7f9d68100680 2023-07-14T18:10:30.639 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:30.639 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:30.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9d500a11a0 msgr2=0x7f9d500a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:30.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9d500a11a0 0x7f9d500a3650 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c010 tx=0x7f9d58005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d68100680 msgr2=0x7f9d6819ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:30.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d68100680 0x7f9d6819ccb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d60007ee0 tx=0x7f9d6000d3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 shutdown_connections 2023-07-14T18:10:30.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9d500a11a0 0x7f9d500a3650 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d68100680 0x7f9d6819ccb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9d680feb00 0x7f9d68068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 --2- 172.21.15.115:0/758065353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680ffcf0 0x7f9d6819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 >> 172.21.15.115:0/758065353 conn(0x7f9d680fa100 msgr2=0x7f9d68103220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:30.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 shutdown_connections 2023-07-14T18:10:30.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.602+0000 7f9d6e26a700 1 -- 172.21.15.115:0/758065353 wait complete. 2023-07-14T18:10:30.653 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:10:15.471213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:10:16.791973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:10:20.663275+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:10:30.654 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:10:30.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 -- 172.21.15.115:0/2305498176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc100690 msgr2=0x7fd3fc106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:30.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 --2- 172.21.15.115:0/2305498176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc100690 0x7fd3fc106f50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd3f400b0a0 tx=0x7fd3f400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 -- 172.21.15.115:0/2305498176 shutdown_connections 2023-07-14T18:10:30.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 --2- 172.21.15.115:0/2305498176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc100690 0x7fd3fc106f50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 --2- 172.21.15.115:0/2305498176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3fc0feb10 0x7fd3fc0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 --2- 172.21.15.115:0/2305498176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3fc0ffd00 0x7fd3fc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 -- 172.21.15.115:0/2305498176 >> 172.21.15.115:0/2305498176 conn(0x7fd3fc0fa130 msgr2=0x7fd3fc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:30.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 -- 172.21.15.115:0/2305498176 shutdown_connections 2023-07-14T18:10:30.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 -- 172.21.15.115:0/2305498176 wait complete. 2023-07-14T18:10:30.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.779+0000 7fd4026a9700 1 Processor -- start 2023-07-14T18:10:30.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd4026a9700 1 -- start start 2023-07-14T18:10:30.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd4026a9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3fc0feb10 0x7fd3fc1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:30.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd4026a9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc0ffd00 0x7fd3fc195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:30.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd4026a9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3fc100690 0x7fd3fc19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:30.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd4026a9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc19d520 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd4026a9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc19d660 con 0x7fd3fc0feb10 2023-07-14T18:10:30.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd4026a9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc19d7f0 con 0x7fd3fc100690 2023-07-14T18:10:30.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fb7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc0ffd00 0x7fd3fc195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:30.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fb7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc0ffd00 0x7fd3fc195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52760/0 (socket says 172.21.15.115:52760) 2023-07-14T18:10:30.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fb7fe700 1 -- 172.21.15.115:0/3260023880 learned_addr learned my addr 172.21.15.115:0/3260023880 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:30.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fbfff700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3fc0feb10 0x7fd3fc1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:30.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd400c46700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3fc100690 0x7fd3fc19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:30.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fb7fe700 1 -- 172.21.15.115:0/3260023880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3fc100690 msgr2=0x7fd3fc19cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:30.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fb7fe700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3fc100690 0x7fd3fc19cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fb7fe700 1 -- 172.21.15.115:0/3260023880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3fc0feb10 msgr2=0x7fd3fc1959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:30.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fb7fe700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3fc0feb10 0x7fd3fc1959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:30.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fb7fe700 1 -- 172.21.15.115:0/3260023880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f4009d00 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd400c46700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3fc100690 0x7fd3fc19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:30.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.780+0000 7fd3fb7fe700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc0ffd00 0x7fd3fc195ee0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000d950 tx=0x7fd3f000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:30.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.781+0000 7fd3f97fa700 1 -- 172.21.15.115:0/3260023880 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0009940 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.781+0000 7fd3f97fa700 1 -- 172.21.15.115:0/3260023880 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd3f0010460 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.781+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc19dad0 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.781+0000 7fd3f97fa700 1 -- 172.21.15.115:0/3260023880 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f000f620 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.781+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc19dfe0 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.782+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc10a1a0 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.782+0000 7fd3f97fa700 1 -- 172.21.15.115:0/3260023880 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fd3f00105d0 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.783+0000 7fd3f97fa700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3e40a1150 0x7fd3e40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:30.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.783+0000 7fd3f97fa700 1 -- 172.21.15.115:0/3260023880 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd3f00cc950 con 0x7fd3fc0ffd00 2023-07-14T18:10:30.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.783+0000 7fd3fbfff700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3e40a1150 0x7fd3e40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:30.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.783+0000 7fd3fbfff700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3e40a1150 0x7fd3e40a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0ffb60 tx=0x7fd3ec00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:30.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.787+0000 7fd3f97fa700 1 -- 172.21.15.115:0/3260023880 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd3f0095eb0 con 0x7fd3fc0ffd00 2023-07-14T18:10:31.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.977+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd3fc19e650 con 0x7fd3fc0ffd00 2023-07-14T18:10:31.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.977+0000 7fd3f97fa700 1 -- 172.21.15.115:0/3260023880 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd3f00160a0 con 0x7fd3fc0ffd00 2023-07-14T18:10:31.016 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:10:31.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.979+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3e40a1150 msgr2=0x7fd3e40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.979+0000 7fd4026a9700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3e40a1150 0x7fd3e40a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0ffb60 tx=0x7fd3ec00b410 comp rx=0 tx=0).stop 2023-07-14T18:10:31.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.979+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc0ffd00 msgr2=0x7fd3fc195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.979+0000 7fd4026a9700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc0ffd00 0x7fd3fc195ee0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000d950 tx=0x7fd3f000dc60 comp rx=0 tx=0).stop 2023-07-14T18:10:31.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.979+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 shutdown_connections 2023-07-14T18:10:31.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.979+0000 7fd4026a9700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3e40a1150 0x7fd3e40a3600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.980+0000 7fd4026a9700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3fc100690 0x7fd3fc19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.980+0000 7fd4026a9700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3fc0feb10 0x7fd3fc1959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.980+0000 7fd4026a9700 1 --2- 172.21.15.115:0/3260023880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3fc0ffd00 0x7fd3fc195ee0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.980+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 >> 172.21.15.115:0/3260023880 conn(0x7fd3fc0fa130 msgr2=0x7fd3fc067860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:31.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.980+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 shutdown_connections 2023-07-14T18:10:31.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:30.980+0000 7fd4026a9700 1 -- 172.21.15.115:0/3260023880 wait complete. 2023-07-14T18:10:31.030 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:10:31.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 -- 172.21.15.115:0/3967731430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc180feb10 msgr2=0x7efc180fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/3967731430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc180feb10 0x7efc180fef20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009990 tx=0x7efc0c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 -- 172.21.15.115:0/3967731430 shutdown_connections 2023-07-14T18:10:31.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/3967731430 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc18100690 0x7efc18106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/3967731430 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc180feb10 0x7efc180fef20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/3967731430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc180ffd00 0x7efc18100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 -- 172.21.15.115:0/3967731430 >> 172.21.15.115:0/3967731430 conn(0x7efc180fa130 msgr2=0x7efc180fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:31.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 -- 172.21.15.115:0/3967731430 shutdown_connections 2023-07-14T18:10:31.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 -- 172.21.15.115:0/3967731430 wait complete. 2023-07-14T18:10:31.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 Processor -- start 2023-07-14T18:10:31.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.116+0000 7efc1e3ca700 1 -- start start 2023-07-14T18:10:31.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1e3ca700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc180feb10 0x7efc18068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1e3ca700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc180ffd00 0x7efc1819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1e3ca700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc18100690 0x7efc1819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1e3ca700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1819d3d0 con 0x7efc18100690 2023-07-14T18:10:31.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1e3ca700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1819d540 con 0x7efc180ffd00 2023-07-14T18:10:31.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1e3ca700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1819d700 con 0x7efc180feb10 2023-07-14T18:10:31.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1c967700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc18100690 0x7efc1819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1c967700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc18100690 0x7efc1819cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52776/0 (socket says 172.21.15.115:52776) 2023-07-14T18:10:31.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1c967700 1 -- 172.21.15.115:0/1129766750 learned_addr learned my addr 172.21.15.115:0/1129766750 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:31.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc17fff700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc180feb10 0x7efc18068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc177fe700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc180ffd00 0x7efc1819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1c967700 1 -- 172.21.15.115:0/1129766750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc180feb10 msgr2=0x7efc18068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1c967700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc180feb10 0x7efc18068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1c967700 1 -- 172.21.15.115:0/1129766750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc180ffd00 msgr2=0x7efc1819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1c967700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc180ffd00 0x7efc1819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1c967700 1 -- 172.21.15.115:0/1129766750 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0c0095f0 con 0x7efc18100690 2023-07-14T18:10:31.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc177fe700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc180ffd00 0x7efc1819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:31.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1c967700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc18100690 0x7efc1819cc30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efc10007ee0 tx=0x7efc1000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:31.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc157fa700 1 -- 172.21.15.115:0/1129766750 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10017070 con 0x7efc18100690 2023-07-14T18:10:31.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1819d9e0 con 0x7efc18100690 2023-07-14T18:10:31.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc157fa700 1 -- 172.21.15.115:0/1129766750 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efc1000f040 con 0x7efc18100690 2023-07-14T18:10:31.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc157fa700 1 -- 172.21.15.115:0/1129766750 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10005710 con 0x7efc18100690 2023-07-14T18:10:31.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.117+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1819def0 con 0x7efc18100690 2023-07-14T18:10:31.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.118+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1810a190 con 0x7efc18100690 2023-07-14T18:10:31.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.119+0000 7efc157fa700 1 -- 172.21.15.115:0/1129766750 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7efc10005870 con 0x7efc18100690 2023-07-14T18:10:31.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.120+0000 7efc157fa700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efc000a1150 0x7efc000a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.120+0000 7efc157fa700 1 -- 172.21.15.115:0/1129766750 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efc100cc9f0 con 0x7efc18100690 2023-07-14T18:10:31.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.120+0000 7efc17fff700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efc000a1150 0x7efc000a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.120+0000 7efc17fff700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efc000a1150 0x7efc000a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc0c006010 tx=0x7efc0c005940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:31.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.123+0000 7efc157fa700 1 -- 172.21.15.115:0/1129766750 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efc10095f50 con 0x7efc18100690 2023-07-14T18:10:31.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.310+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efc1810a350 con 0x7efc18100690 2023-07-14T18:10:31.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.310+0000 7efc157fa700 1 -- 172.21.15.115:0/1129766750 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7efc10099570 con 0x7efc18100690 2023-07-14T18:10:31.349 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:10:31.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.312+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efc000a1150 msgr2=0x7efc000a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.312+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efc000a1150 0x7efc000a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc0c006010 tx=0x7efc0c005940 comp rx=0 tx=0).stop 2023-07-14T18:10:31.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.312+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc18100690 msgr2=0x7efc1819cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.312+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc18100690 0x7efc1819cc30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efc10007ee0 tx=0x7efc1000d3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.313+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 shutdown_connections 2023-07-14T18:10:31.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.313+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efc000a1150 0x7efc000a3600 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.313+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efc18100690 0x7efc1819cc30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.313+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc180feb10 0x7efc18068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.313+0000 7efc1e3ca700 1 --2- 172.21.15.115:0/1129766750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc180ffd00 0x7efc1819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.313+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 >> 172.21.15.115:0/1129766750 conn(0x7efc180fa130 msgr2=0x7efc180fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:31.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.313+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 shutdown_connections 2023-07-14T18:10:31.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.313+0000 7efc1e3ca700 1 -- 172.21.15.115:0/1129766750 wait complete. 2023-07-14T18:10:31.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:30 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/758065353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:31.363 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:10:31.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:30 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/758065353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:31.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:30 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/758065353' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:31.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.448+0000 7f7a775ce700 1 -- 172.21.15.115:0/1761056685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a700ffc10 msgr2=0x7f7a70100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.448+0000 7f7a775ce700 1 --2- 172.21.15.115:0/1761056685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a700ffc10 0x7f7a70100020 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009990 tx=0x7f7a6c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.448+0000 7f7a775ce700 1 -- 172.21.15.115:0/1761056685 shutdown_connections 2023-07-14T18:10:31.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.448+0000 7f7a775ce700 1 --2- 172.21.15.115:0/1761056685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70101790 0x7f7a70108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.448+0000 7f7a775ce700 1 --2- 172.21.15.115:0/1761056685 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a700ffc10 0x7f7a70100020 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.448+0000 7f7a775ce700 1 --2- 172.21.15.115:0/1761056685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a70100e00 0x7f7a70101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.448+0000 7f7a775ce700 1 -- 172.21.15.115:0/1761056685 >> 172.21.15.115:0/1761056685 conn(0x7f7a700fb250 msgr2=0x7f7a700fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:31.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.448+0000 7f7a775ce700 1 -- 172.21.15.115:0/1761056685 shutdown_connections 2023-07-14T18:10:31.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.448+0000 7f7a775ce700 1 -- 172.21.15.115:0/1761056685 wait complete. 2023-07-14T18:10:31.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a775ce700 1 Processor -- start 2023-07-14T18:10:31.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a775ce700 1 -- start start 2023-07-14T18:10:31.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a775ce700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a700ffc10 0x7f7a70195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a775ce700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a70100e00 0x7f7a70195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a775ce700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70101790 0x7f7a7019ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a775ce700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7019d3e0 con 0x7f7a70100e00 2023-07-14T18:10:31.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a775ce700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7019d520 con 0x7f7a70101790 2023-07-14T18:10:31.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a775ce700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7019d6b0 con 0x7f7a700ffc10 2023-07-14T18:10:31.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a74b69700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a70100e00 0x7f7a70195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.490 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a74b69700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a70100e00 0x7f7a70195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52788/0 (socket says 172.21.15.115:52788) 2023-07-14T18:10:31.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a74b69700 1 -- 172.21.15.115:0/3353260490 learned_addr learned my addr 172.21.15.115:0/3353260490 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:31.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a75b6b700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70101790 0x7f7a7019ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a74b69700 1 -- 172.21.15.115:0/3353260490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a700ffc10 msgr2=0x7f7a70195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a74b69700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a700ffc10 0x7f7a70195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a74b69700 1 -- 172.21.15.115:0/3353260490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70101790 msgr2=0x7f7a7019ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a74b69700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70101790 0x7f7a7019ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a74b69700 1 -- 172.21.15.115:0/3353260490 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c0095f0 con 0x7f7a70100e00 2023-07-14T18:10:31.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a74b69700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a70100e00 0x7f7a70195da0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000c920 tx=0x7f7a6000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:31.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.449+0000 7f7a667fc700 1 -- 172.21.15.115:0/3353260490 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60007b00 con 0x7f7a70100e00 2023-07-14T18:10:31.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.450+0000 7f7a667fc700 1 -- 172.21.15.115:0/3353260490 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7a60004cc0 con 0x7f7a70100e00 2023-07-14T18:10:31.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.450+0000 7f7a667fc700 1 -- 172.21.15.115:0/3353260490 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60005590 con 0x7f7a70100e00 2023-07-14T18:10:31.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.450+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7019d990 con 0x7f7a70100e00 2023-07-14T18:10:31.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.450+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7019dea0 con 0x7f7a70100e00 2023-07-14T18:10:31.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.450+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7010b290 con 0x7f7a70100e00 2023-07-14T18:10:31.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.451+0000 7f7a667fc700 1 -- 172.21.15.115:0/3353260490 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7a60004750 con 0x7f7a70100e00 2023-07-14T18:10:31.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.452+0000 7f7a667fc700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.452+0000 7f7a667fc700 1 -- 172.21.15.115:0/3353260490 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7a600cca50 con 0x7f7a70100e00 2023-07-14T18:10:31.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.452+0000 7f7a7536a700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.453+0000 7f7a7536a700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00b5c0 tx=0x7f7a6c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:31.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.455+0000 7f7a667fc700 1 -- 172.21.15.115:0/3353260490 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7a60096030 con 0x7f7a70100e00 2023-07-14T18:10:31.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.645+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7a70106bb0 con 0x7f7a70100e00 2023-07-14T18:10:31.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.645+0000 7f7a667fc700 1 -- 172.21.15.115:0/3353260490 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7a6001d070 con 0x7f7a70100e00 2023-07-14T18:10:31.684 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 64 2023-07-14T18:10:31.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a5c0a11d0 msgr2=0x7f7a5c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00b5c0 tx=0x7f7a6c005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a70100e00 msgr2=0x7f7a70195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a70100e00 0x7f7a70195da0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000c920 tx=0x7f7a6000cce0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 shutdown_connections 2023-07-14T18:10:31.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f7a5c0a11d0 0x7f7a5c0a3680 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70101790 0x7f7a7019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a700ffc10 0x7f7a70195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 --2- 172.21.15.115:0/3353260490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7a70100e00 0x7f7a70195da0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 >> 172.21.15.115:0/3353260490 conn(0x7f7a700fb250 msgr2=0x7f7a700fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:31.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 shutdown_connections 2023-07-14T18:10:31.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.647+0000 7f7a775ce700 1 -- 172.21.15.115:0/3353260490 wait complete. 2023-07-14T18:10:31.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:10:31.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.783+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/117720521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00ffd00 msgr2=0x7fd2b0100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.783+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/117720521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00ffd00 0x7fd2b0100150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009990 tx=0x7fd29c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.783+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/117720521 shutdown_connections 2023-07-14T18:10:31.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.783+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/117720521 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2b0100690 0x7fd2b0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.783+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/117720521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b00feb10 0x7fd2b00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.783+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/117720521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00ffd00 0x7fd2b0100150 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.783+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/117720521 >> 172.21.15.115:0/117720521 conn(0x7fd2b00fa130 msgr2=0x7fd2b00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:31.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.783+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/117720521 shutdown_connections 2023-07-14T18:10:31.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.783+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/117720521 wait complete. 2023-07-14T18:10:31.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b6f8a700 1 Processor -- start 2023-07-14T18:10:31.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b6f8a700 1 -- start start 2023-07-14T18:10:31.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b6f8a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00feb10 0x7fd2b0068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b6f8a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2b00ffd00 0x7fd2b019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b6f8a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b0100690 0x7fd2b019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b6f8a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b019d3f0 con 0x7fd2b00feb10 2023-07-14T18:10:31.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b6f8a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b019d560 con 0x7fd2b0100690 2023-07-14T18:10:31.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b6f8a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b019d720 con 0x7fd2b00ffd00 2023-07-14T18:10:31.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b4d26700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00feb10 0x7fd2b0068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b4d26700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00feb10 0x7fd2b0068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52800/0 (socket says 172.21.15.115:52800) 2023-07-14T18:10:31.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b4d26700 1 -- 172.21.15.115:0/3075666622 learned_addr learned my addr 172.21.15.115:0/3075666622 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:31.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2affff700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2b00ffd00 0x7fd2b019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.784+0000 7fd2b5527700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b0100690 0x7fd2b019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2b4d26700 1 -- 172.21.15.115:0/3075666622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2b00ffd00 msgr2=0x7fd2b019a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2b4d26700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2b00ffd00 0x7fd2b019a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2b4d26700 1 -- 172.21.15.115:0/3075666622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b0100690 msgr2=0x7fd2b019cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:31.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2b4d26700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b0100690 0x7fd2b019cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:31.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2b4d26700 1 -- 172.21.15.115:0/3075666622 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c0095f0 con 0x7fd2b00feb10 2023-07-14T18:10:31.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2b4d26700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00feb10 0x7fd2b0068310 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400d980 tx=0x7fd2a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:31.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2adffb700 1 -- 172.21.15.115:0/3075666622 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4004020 con 0x7fd2b00feb10 2023-07-14T18:10:31.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2adffb700 1 -- 172.21.15.115:0/3075666622 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd2a4009730 con 0x7fd2b00feb10 2023-07-14T18:10:31.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2adffb700 1 -- 172.21.15.115:0/3075666622 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4003a00 con 0x7fd2b00feb10 2023-07-14T18:10:31.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b019da00 con 0x7fd2b00feb10 2023-07-14T18:10:31.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.785+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b019df10 con 0x7fd2b00feb10 2023-07-14T18:10:31.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.786+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b010a190 con 0x7fd2b00feb10 2023-07-14T18:10:31.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.787+0000 7fd2adffb700 1 -- 172.21.15.115:0/3075666622 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fd2a40098a0 con 0x7fd2b00feb10 2023-07-14T18:10:31.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.787+0000 7fd2adffb700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2a00a11d0 0x7fd2a00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:31.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.787+0000 7fd2affff700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2a00a11d0 0x7fd2a00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:31.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.787+0000 7fd2adffb700 1 -- 172.21.15.115:0/3075666622 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd2a40cc730 con 0x7fd2b00feb10 2023-07-14T18:10:31.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.788+0000 7fd2affff700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2a00a11d0 0x7fd2a00a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00c010 tx=0x7fd29c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:31.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.792+0000 7fd2adffb700 1 -- 172.21.15.115:0/3075666622 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd2a4095c90 con 0x7fd2b00feb10 2023-07-14T18:10:32.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.981+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd2b010a350 con 0x7fd2b00feb10 2023-07-14T18:10:32.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.981+0000 7fd2adffb700 1 -- 172.21.15.115:0/3075666622 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd2a40992b0 con 0x7fd2b00feb10 2023-07-14T18:10:32.020 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:10:32.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2a00a11d0 msgr2=0x7fd2a00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2a00a11d0 0x7fd2a00a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00c010 tx=0x7fd29c005960 comp rx=0 tx=0).stop 2023-07-14T18:10:32.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00feb10 msgr2=0x7fd2b0068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00feb10 0x7fd2b0068310 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400d980 tx=0x7fd2a400dc90 comp rx=0 tx=0).stop 2023-07-14T18:10:32.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 shutdown_connections 2023-07-14T18:10:32.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2a00a11d0 0x7fd2a00a3680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b0100690 0x7fd2b019cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2b00feb10 0x7fd2b0068310 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 --2- 172.21.15.115:0/3075666622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2b00ffd00 0x7fd2b019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 >> 172.21.15.115:0/3075666622 conn(0x7fd2b00fa130 msgr2=0x7fd2b00fc460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:32.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 shutdown_connections 2023-07-14T18:10:32.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:31.984+0000 7fd2b6f8a700 1 -- 172.21.15.115:0/3075666622 wait complete. 2023-07-14T18:10:32.034 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T18:10:32.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1026252524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f041006c0 msgr2=0x7f1f04106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1026252524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f041006c0 0x7f1f04106f80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00b0a0 tx=0x7f1efc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1026252524 shutdown_connections 2023-07-14T18:10:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1026252524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f041006c0 0x7f1f04106f80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1026252524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f040feb40 0x7f1f040fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1026252524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f040ffd30 0x7f1f04100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1026252524 >> 172.21.15.115:0/1026252524 conn(0x7f1f040fa160 msgr2=0x7f1f040fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:32.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1026252524 shutdown_connections 2023-07-14T18:10:32.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1026252524 wait complete. 2023-07-14T18:10:32.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 Processor -- start 2023-07-14T18:10:32.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 -- start start 2023-07-14T18:10:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.117+0000 7f1f0c4fd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f040feb40 0x7f1f041947d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0c4fd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f040ffd30 0x7f1f04194d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0c4fd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f041006c0 0x7f1f0419bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0c4fd700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0419c350 con 0x7f1f040feb40 2023-07-14T18:10:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0c4fd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0419c490 con 0x7f1f040ffd30 2023-07-14T18:10:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0c4fd700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0419c620 con 0x7f1f041006c0 2023-07-14T18:10:32.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0a299700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f040feb40 0x7f1f041947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0a299700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f040feb40 0x7f1f041947d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52814/0 (socket says 172.21.15.115:52814) 2023-07-14T18:10:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0a299700 1 -- 172.21.15.115:0/1822054022 learned_addr learned my addr 172.21.15.115:0/1822054022 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0aa9a700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f041006c0 0x7f1f0419bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f09a98700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f040ffd30 0x7f1f04194d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0a299700 1 -- 172.21.15.115:0/1822054022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f041006c0 msgr2=0x7f1f0419bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0a299700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f041006c0 0x7f1f0419bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0a299700 1 -- 172.21.15.115:0/1822054022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f040ffd30 msgr2=0x7f1f04194d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0a299700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f040ffd30 0x7f1f04194d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0a299700 1 -- 172.21.15.115:0/1822054022 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc009d00 con 0x7f1f040feb40 2023-07-14T18:10:32.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f09a98700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f040ffd30 0x7f1f04194d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:32.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0a299700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f040feb40 0x7f1f041947d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400c990 tx=0x7f1ef400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:32.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1efb7fe700 1 -- 172.21.15.115:0/1822054022 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4007b00 con 0x7f1f040feb40 2023-07-14T18:10:32.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0419c900 con 0x7f1f040feb40 2023-07-14T18:10:32.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1efb7fe700 1 -- 172.21.15.115:0/1822054022 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1ef4007c60 con 0x7f1f040feb40 2023-07-14T18:10:32.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1efb7fe700 1 -- 172.21.15.115:0/1822054022 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4018420 con 0x7f1f040feb40 2023-07-14T18:10:32.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.118+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0419ce10 con 0x7f1f040feb40 2023-07-14T18:10:32.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.119+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f04106030 con 0x7f1f040feb40 2023-07-14T18:10:32.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.120+0000 7f1efb7fe700 1 -- 172.21.15.115:0/1822054022 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f1ef400f450 con 0x7f1f040feb40 2023-07-14T18:10:32.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.121+0000 7f1efb7fe700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1ef00a1150 0x7f1ef00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.121+0000 7f1efb7fe700 1 -- 172.21.15.115:0/1822054022 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1ef40ccc20 con 0x7f1f040feb40 2023-07-14T18:10:32.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.121+0000 7f1f09a98700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1ef00a1150 0x7f1ef00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.121+0000 7f1f09a98700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1ef00a1150 0x7f1ef00a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1f00005fd0 tx=0x7f1f00005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:32.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.124+0000 7f1efb7fe700 1 -- 172.21.15.115:0/1822054022 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1ef4096210 con 0x7f1f040feb40 2023-07-14T18:10:32.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:31 smithi115 ceph-mon[127451]: pgmap v258: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 121 B/s wr, 2 op/s 2023-07-14T18:10:32.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:31 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3260023880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:10:32.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:31 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1129766750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:10:32.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:31 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3353260490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:10:32.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.330+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1f0402c8e0 con 0x7f1f040feb40 2023-07-14T18:10:32.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.330+0000 7f1efb7fe700 1 -- 172.21.15.115:0/1822054022 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+600 (secure 0 0 0) 0x7f1ef400f700 con 0x7f1f040feb40 2023-07-14T18:10:32.369 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6800/2169355495 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:32.370 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/4031581902 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:32.370 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/3311800921 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:32.370 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6801/2169355495 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:32.370 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2790161712 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:32.370 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2625519889 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:32.370 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6801/568655687 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:32.370 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/3340901626 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:32.370 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6800/568655687 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:32.371 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2416566340 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:32.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:31 smithi150 ceph-mon[128853]: pgmap v258: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 121 B/s wr, 2 op/s 2023-07-14T18:10:32.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:31 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3260023880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:10:32.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:31 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1129766750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:10:32.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:31 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3353260490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:10:32.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1ef00a1150 msgr2=0x7f1ef00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1ef00a1150 0x7f1ef00a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1f00005fd0 tx=0x7f1f00005ee0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f040feb40 msgr2=0x7f1f041947d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f040feb40 0x7f1f041947d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400c990 tx=0x7f1ef400cca0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 shutdown_connections 2023-07-14T18:10:32.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1ef00a1150 0x7f1ef00a3600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f041006c0 0x7f1f0419bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1f040feb40 0x7f1f041947d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 --2- 172.21.15.115:0/1822054022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f040ffd30 0x7f1f04194d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 >> 172.21.15.115:0/1822054022 conn(0x7f1f040fa160 msgr2=0x7f1f04103170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:32.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 shutdown_connections 2023-07-14T18:10:32.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.333+0000 7f1f0c4fd700 1 -- 172.21.15.115:0/1822054022 wait complete. 2023-07-14T18:10:32.374 INFO:teuthology.orchestra.run.smithi115.stderr:listed 10 entries 2023-07-14T18:10:32.383 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:10:32.383 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:10:32.401 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:10:32.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:31 smithi187 ceph-mon[129317]: pgmap v258: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 121 B/s wr, 2 op/s 2023-07-14T18:10:32.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:31 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3260023880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:10:32.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:31 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1129766750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:10:32.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:31 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3353260490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:10:32.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 -- 172.21.15.115:0/772101895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0ffd40 msgr2=0x7fbccc100190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 --2- 172.21.15.115:0/772101895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0ffd40 0x7fbccc100190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc009990 tx=0x7fbcbc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 -- 172.21.15.115:0/772101895 shutdown_connections 2023-07-14T18:10:32.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 --2- 172.21.15.115:0/772101895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc1006d0 0x7fbccc106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 --2- 172.21.15.115:0/772101895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbccc0feb50 0x7fbccc0fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 --2- 172.21.15.115:0/772101895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0ffd40 0x7fbccc100190 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 -- 172.21.15.115:0/772101895 >> 172.21.15.115:0/772101895 conn(0x7fbccc0fa150 msgr2=0x7fbccc0fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:32.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 -- 172.21.15.115:0/772101895 shutdown_connections 2023-07-14T18:10:32.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 -- 172.21.15.115:0/772101895 wait complete. 2023-07-14T18:10:32.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.524+0000 7fbcd198e700 1 Processor -- start 2023-07-14T18:10:32.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbcd198e700 1 -- start start 2023-07-14T18:10:32.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbcd198e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0feb50 0x7fbccc1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbcd198e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbccc0ffd40 0x7fbccc197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbcd198e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc1006d0 0x7fbccc19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbcd198e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc19f540 con 0x7fbccc0feb50 2023-07-14T18:10:32.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbcd198e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc19f680 con 0x7fbccc1006d0 2023-07-14T18:10:32.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbcd198e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc19f810 con 0x7fbccc0ffd40 2023-07-14T18:10:32.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccaffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0feb50 0x7fbccc1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccaffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0feb50 0x7fbccc1979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52828/0 (socket says 172.21.15.115:52828) 2023-07-14T18:10:32.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccaffd700 1 -- 172.21.15.115:0/678608115 learned_addr learned my addr 172.21.15.115:0/678608115 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:32.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccb7fe700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc1006d0 0x7fbccc19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccaffd700 1 -- 172.21.15.115:0/678608115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbccc0ffd40 msgr2=0x7fbccc197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccaffd700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbccc0ffd40 0x7fbccc197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccaffd700 1 -- 172.21.15.115:0/678608115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc1006d0 msgr2=0x7fbccc19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccaffd700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc1006d0 0x7fbccc19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccaffd700 1 -- 172.21.15.115:0/678608115 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc0095f0 con 0x7fbccc0feb50 2023-07-14T18:10:32.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.525+0000 7fbccaffd700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0feb50 0x7fbccc1979c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400d980 tx=0x7fbcb400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:32.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.526+0000 7fbcd098c700 1 -- 172.21.15.115:0/678608115 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb400de60 con 0x7fbccc0feb50 2023-07-14T18:10:32.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.526+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc19faf0 con 0x7fbccc0feb50 2023-07-14T18:10:32.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.526+0000 7fbcd098c700 1 -- 172.21.15.115:0/678608115 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbcb40044b0 con 0x7fbccc0feb50 2023-07-14T18:10:32.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.526+0000 7fbcd098c700 1 -- 172.21.15.115:0/678608115 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4003b40 con 0x7fbccc0feb50 2023-07-14T18:10:32.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.526+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc1a0050 con 0x7fbccc0feb50 2023-07-14T18:10:32.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.527+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc10a1d0 con 0x7fbccc0feb50 2023-07-14T18:10:32.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.528+0000 7fbcd098c700 1 -- 172.21.15.115:0/678608115 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fbcb4003ca0 con 0x7fbccc0feb50 2023-07-14T18:10:32.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.529+0000 7fbcd098c700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcb80a11a0 0x7fbcb80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.529+0000 7fbcd098c700 1 -- 172.21.15.115:0/678608115 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbcb40ccb70 con 0x7fbccc0feb50 2023-07-14T18:10:32.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.529+0000 7fbcca7fc700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcb80a11a0 0x7fbcb80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.529+0000 7fbcca7fc700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcb80a11a0 0x7fbcb80a3650 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc00b5c0 tx=0x7fbcbc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:32.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.532+0000 7fbcd098c700 1 -- 172.21.15.115:0/678608115 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbcb40960d0 con 0x7fbccc0feb50 2023-07-14T18:10:32.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.723+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbccc10a3c0 con 0x7fbccc0feb50 2023-07-14T18:10:32.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.724+0000 7fbcd098c700 1 -- 172.21.15.115:0/678608115 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7fbcb4009940 con 0x7fbccc0feb50 2023-07-14T18:10:32.763 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:32.764 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:32.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcb80a11a0 msgr2=0x7fbcb80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcb80a11a0 0x7fbcb80a3650 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc00b5c0 tx=0x7fbcbc005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0feb50 msgr2=0x7fbccc1979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0feb50 0x7fbccc1979c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400d980 tx=0x7fbcb400dd40 comp rx=0 tx=0).stop 2023-07-14T18:10:32.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 shutdown_connections 2023-07-14T18:10:32.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcb80a11a0 0x7fbcb80a3650 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbccc1006d0 0x7fbccc19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbccc0feb50 0x7fbccc1979c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 --2- 172.21.15.115:0/678608115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbccc0ffd40 0x7fbccc197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.726+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 >> 172.21.15.115:0/678608115 conn(0x7fbccc0fa150 msgr2=0x7fbccc068ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:32.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.727+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 shutdown_connections 2023-07-14T18:10:32.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.727+0000 7fbcd198e700 1 -- 172.21.15.115:0/678608115 wait complete. 2023-07-14T18:10:32.778 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:10:15.471213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:10:16.791973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:10:20.663275+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:10:32.779 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:10:32.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.865+0000 7ff5097a8700 1 -- 172.21.15.115:0/4045705882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5041018e0 msgr2=0x7ff5041081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.865+0000 7ff5097a8700 1 --2- 172.21.15.115:0/4045705882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5041018e0 0x7ff5041081a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00b0a0 tx=0x7ff4fc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.865+0000 7ff5097a8700 1 -- 172.21.15.115:0/4045705882 shutdown_connections 2023-07-14T18:10:32.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.865+0000 7ff5097a8700 1 --2- 172.21.15.115:0/4045705882 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5041018e0 0x7ff5041081a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.865+0000 7ff5097a8700 1 --2- 172.21.15.115:0/4045705882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff5040fdf80 0x7ff5040fe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.865+0000 7ff5097a8700 1 --2- 172.21.15.115:0/4045705882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff504100f50 0x7ff5041013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.865+0000 7ff5097a8700 1 -- 172.21.15.115:0/4045705882 >> 172.21.15.115:0/4045705882 conn(0x7ff5040f9980 msgr2=0x7ff5040fbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:32.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.865+0000 7ff5097a8700 1 -- 172.21.15.115:0/4045705882 shutdown_connections 2023-07-14T18:10:32.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.865+0000 7ff5097a8700 1 -- 172.21.15.115:0/4045705882 wait complete. 2023-07-14T18:10:32.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff5097a8700 1 Processor -- start 2023-07-14T18:10:32.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff5097a8700 1 -- start start 2023-07-14T18:10:32.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff5097a8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5040fdf80 0x7ff504195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff5097a8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff504100f50 0x7ff504195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff5097a8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5041018e0 0x7ff50419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff5097a8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50419d5a0 con 0x7ff5040fdf80 2023-07-14T18:10:32.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff5097a8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50419d6e0 con 0x7ff5041018e0 2023-07-14T18:10:32.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff5097a8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50419d870 con 0x7ff504100f50 2023-07-14T18:10:32.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff502ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5040fdf80 0x7ff504195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.866+0000 7ff502ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5040fdf80 0x7ff504195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52842/0 (socket says 172.21.15.115:52842) 2023-07-14T18:10:32.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff502ffd700 1 -- 172.21.15.115:0/2974835924 learned_addr learned my addr 172.21.15.115:0/2974835924 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:32.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff5037fe700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5041018e0 0x7ff50419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff5027fc700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff504100f50 0x7ff504195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff502ffd700 1 -- 172.21.15.115:0/2974835924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff504100f50 msgr2=0x7ff504195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff502ffd700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff504100f50 0x7ff504195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff502ffd700 1 -- 172.21.15.115:0/2974835924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5041018e0 msgr2=0x7ff50419cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:32.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff502ffd700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5041018e0 0x7ff50419cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:32.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff502ffd700 1 -- 172.21.15.115:0/2974835924 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc009d00 con 0x7ff5040fdf80 2023-07-14T18:10:32.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff5027fc700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff504100f50 0x7ff504195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:32.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff5037fe700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5041018e0 0x7ff50419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:32.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff502ffd700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5040fdf80 0x7ff504195a20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400b500 tx=0x7ff4f400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:32.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff4ebfff700 1 -- 172.21.15.115:0/2974835924 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4011830 con 0x7ff5040fdf80 2023-07-14T18:10:32.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50419db50 con 0x7ff5040fdf80 2023-07-14T18:10:32.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff4ebfff700 1 -- 172.21.15.115:0/2974835924 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff4f4011e20 con 0x7ff5040fdf80 2023-07-14T18:10:32.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff4ebfff700 1 -- 172.21.15.115:0/2974835924 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4010dd0 con 0x7ff5040fdf80 2023-07-14T18:10:32.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.867+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50419e060 con 0x7ff5040fdf80 2023-07-14T18:10:32.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.868+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff504107360 con 0x7ff5040fdf80 2023-07-14T18:10:32.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.869+0000 7ff4ebfff700 1 -- 172.21.15.115:0/2974835924 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ff4f4011990 con 0x7ff5040fdf80 2023-07-14T18:10:32.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.870+0000 7ff4ebfff700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4ec0a1150 0x7ff4ec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:32.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.870+0000 7ff5027fc700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4ec0a1150 0x7ff4ec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:32.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.870+0000 7ff4ebfff700 1 -- 172.21.15.115:0/2974835924 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff4f40cc5d0 con 0x7ff5040fdf80 2023-07-14T18:10:32.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.870+0000 7ff5027fc700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4ec0a1150 0x7ff4ec0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009be0 tx=0x7ff4f8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:32.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:32.874+0000 7ff4ebfff700 1 -- 172.21.15.115:0/2974835924 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff4f4095b30 con 0x7ff5040fdf80 2023-07-14T18:10:33.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.064+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff504061390 con 0x7ff5040fdf80 2023-07-14T18:10:33.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.064+0000 7ff4ebfff700 1 -- 172.21.15.115:0/2974835924 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff4f4099150 con 0x7ff5040fdf80 2023-07-14T18:10:33.103 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:10:33.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4ec0a1150 msgr2=0x7ff4ec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4ec0a1150 0x7ff4ec0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009be0 tx=0x7ff4f8009260 comp rx=0 tx=0).stop 2023-07-14T18:10:33.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5040fdf80 msgr2=0x7ff504195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5040fdf80 0x7ff504195a20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400b500 tx=0x7ff4f400b8c0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 shutdown_connections 2023-07-14T18:10:33.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4ec0a1150 0x7ff4ec0a3600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5041018e0 0x7ff50419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff5040fdf80 0x7ff504195a20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 --2- 172.21.15.115:0/2974835924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff504100f50 0x7ff504195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.066+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 >> 172.21.15.115:0/2974835924 conn(0x7ff5040f9980 msgr2=0x7ff5041044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:33.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.067+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 shutdown_connections 2023-07-14T18:10:33.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.067+0000 7ff5097a8700 1 -- 172.21.15.115:0/2974835924 wait complete. 2023-07-14T18:10:33.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:32 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3075666622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:10:33.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:32 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1822054022' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:10:33.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:32 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/678608115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:33.117 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:10:33.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.203+0000 7fba8edc6700 1 -- 172.21.15.115:0/2702768443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba880ff6b0 msgr2=0x7fba88106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.203+0000 7fba8edc6700 1 --2- 172.21.15.115:0/2702768443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba880ff6b0 0x7fba88106f60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fba8000b0a0 tx=0x7fba8000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.203+0000 7fba8edc6700 1 -- 172.21.15.115:0/2702768443 shutdown_connections 2023-07-14T18:10:33.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.203+0000 7fba8edc6700 1 --2- 172.21.15.115:0/2702768443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba880ff6b0 0x7fba88106f60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.203+0000 7fba8edc6700 1 --2- 172.21.15.115:0/2702768443 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba881000b0 0x7fba881004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.203+0000 7fba8edc6700 1 --2- 172.21.15.115:0/2702768443 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba880fed00 0x7fba880ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.203+0000 7fba8edc6700 1 -- 172.21.15.115:0/2702768443 >> 172.21.15.115:0/2702768443 conn(0x7fba880f98a0 msgr2=0x7fba880fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:33.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.203+0000 7fba8edc6700 1 -- 172.21.15.115:0/2702768443 shutdown_connections 2023-07-14T18:10:33.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.203+0000 7fba8edc6700 1 -- 172.21.15.115:0/2702768443 wait complete. 2023-07-14T18:10:33.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8edc6700 1 Processor -- start 2023-07-14T18:10:33.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8edc6700 1 -- start start 2023-07-14T18:10:33.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8edc6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba880fed00 0x7fba880fe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8edc6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba880ff6b0 0x7fba8819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8edc6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba881000b0 0x7fba8819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8edc6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8819d510 con 0x7fba881000b0 2023-07-14T18:10:33.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8edc6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8819d680 con 0x7fba880fed00 2023-07-14T18:10:33.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8edc6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8819d840 con 0x7fba880ff6b0 2023-07-14T18:10:33.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8d363700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba881000b0 0x7fba8819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8cb62700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba880fed00 0x7fba880fe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8d363700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba881000b0 0x7fba8819cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52860/0 (socket says 172.21.15.115:52860) 2023-07-14T18:10:33.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8d363700 1 -- 172.21.15.115:0/3560423453 learned_addr learned my addr 172.21.15.115:0/3560423453 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:33.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba87fff700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba880ff6b0 0x7fba8819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8d363700 1 -- 172.21.15.115:0/3560423453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba880ff6b0 msgr2=0x7fba8819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8d363700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba880ff6b0 0x7fba8819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8d363700 1 -- 172.21.15.115:0/3560423453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba880fed00 msgr2=0x7fba880fe7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8d363700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba880fed00 0x7fba880fe7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.204+0000 7fba8d363700 1 -- 172.21.15.115:0/3560423453 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba80009d00 con 0x7fba881000b0 2023-07-14T18:10:33.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.205+0000 7fba8d363700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba881000b0 0x7fba8819cd70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fba8000b070 tx=0x7fba80004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:33.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.205+0000 7fba85ffb700 1 -- 172.21.15.115:0/3560423453 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8000e040 con 0x7fba881000b0 2023-07-14T18:10:33.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.205+0000 7fba85ffb700 1 -- 172.21.15.115:0/3560423453 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fba8001baf0 con 0x7fba881000b0 2023-07-14T18:10:33.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.205+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8819dac0 con 0x7fba881000b0 2023-07-14T18:10:33.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.205+0000 7fba85ffb700 1 -- 172.21.15.115:0/3560423453 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba80005760 con 0x7fba881000b0 2023-07-14T18:10:33.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.205+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8819df90 con 0x7fba881000b0 2023-07-14T18:10:33.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.206+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8810a1e0 con 0x7fba881000b0 2023-07-14T18:10:33.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.207+0000 7fba85ffb700 1 -- 172.21.15.115:0/3560423453 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fba8001bc60 con 0x7fba881000b0 2023-07-14T18:10:33.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.207+0000 7fba85ffb700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fba780a11d0 0x7fba780a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.208+0000 7fba85ffb700 1 -- 172.21.15.115:0/3560423453 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fba800ce820 con 0x7fba881000b0 2023-07-14T18:10:33.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.208+0000 7fba8cb62700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fba780a11d0 0x7fba780a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.208+0000 7fba8cb62700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fba780a11d0 0x7fba780a3680 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009a10 tx=0x7fba7c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:33.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.213+0000 7fba85ffb700 1 -- 172.21.15.115:0/3560423453 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fba80097e10 con 0x7fba881000b0 2023-07-14T18:10:33.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:32 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3075666622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:10:33.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:32 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1822054022' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:10:33.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:32 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/678608115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:33.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:32 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3075666622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:10:33.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:32 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1822054022' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:10:33.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:32 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/678608115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:33.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.401+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba88105ab0 con 0x7fba881000b0 2023-07-14T18:10:33.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.401+0000 7fba85ffb700 1 -- 172.21.15.115:0/3560423453 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fba8009b430 con 0x7fba881000b0 2023-07-14T18:10:33.440 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:10:33.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fba780a11d0 msgr2=0x7fba780a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fba780a11d0 0x7fba780a3680 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009a10 tx=0x7fba7c009190 comp rx=0 tx=0).stop 2023-07-14T18:10:33.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba881000b0 msgr2=0x7fba8819cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba881000b0 0x7fba8819cd70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fba8000b070 tx=0x7fba80004d00 comp rx=0 tx=0).stop 2023-07-14T18:10:33.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 shutdown_connections 2023-07-14T18:10:33.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fba780a11d0 0x7fba780a3680 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba881000b0 0x7fba8819cd70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba880fed00 0x7fba880fe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 --2- 172.21.15.115:0/3560423453 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba880ff6b0 0x7fba8819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.403+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 >> 172.21.15.115:0/3560423453 conn(0x7fba880f98a0 msgr2=0x7fba880fbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:33.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.404+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 shutdown_connections 2023-07-14T18:10:33.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.404+0000 7fba8edc6700 1 -- 172.21.15.115:0/3560423453 wait complete. 2023-07-14T18:10:33.454 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:10:33.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3244376961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 msgr2=0x7ff4e80fe2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3244376961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 0x7ff4e80fe2f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009990 tx=0x7ff4dc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3244376961 shutdown_connections 2023-07-14T18:10:33.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3244376961 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4e81018e0 0x7ff4e81081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3244376961 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 0x7ff4e80fe2f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3244376961 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e8100f50 0x7ff4e81013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3244376961 >> 172.21.15.115:0/3244376961 conn(0x7ff4e80f98a0 msgr2=0x7ff4e80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:33.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3244376961 shutdown_connections 2023-07-14T18:10:33.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3244376961 wait complete. 2023-07-14T18:10:33.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.539+0000 7ff4ed4b7700 1 Processor -- start 2023-07-14T18:10:33.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4ed4b7700 1 -- start start 2023-07-14T18:10:33.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4ed4b7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 0x7ff4e81008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4ed4b7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4e8100f50 0x7ff4e80fef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4ed4b7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e81018e0 0x7ff4e80ff440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4ed4b7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e80ffbe0 con 0x7ff4e80fdee0 2023-07-14T18:10:33.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4ed4b7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e80ffd50 con 0x7ff4e81018e0 2023-07-14T18:10:33.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4ed4b7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e819f950 con 0x7ff4e8100f50 2023-07-14T18:10:33.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 0x7ff4e81008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 0x7ff4e81008c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52872/0 (socket says 172.21.15.115:52872) 2023-07-14T18:10:33.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e6ffd700 1 -- 172.21.15.115:0/3493717982 learned_addr learned my addr 172.21.15.115:0/3493717982 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:33.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e77fe700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e81018e0 0x7ff4e80ff440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e67fc700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4e8100f50 0x7ff4e80fef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e6ffd700 1 -- 172.21.15.115:0/3493717982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4e8100f50 msgr2=0x7ff4e80fef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e6ffd700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4e8100f50 0x7ff4e80fef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e6ffd700 1 -- 172.21.15.115:0/3493717982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e81018e0 msgr2=0x7ff4e80ff440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e6ffd700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e81018e0 0x7ff4e80ff440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e6ffd700 1 -- 172.21.15.115:0/3493717982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc0095f0 con 0x7ff4e80fdee0 2023-07-14T18:10:33.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4e6ffd700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 0x7ff4e81008c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc004730 tx=0x7ff4dc004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:33.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.540+0000 7ff4cffff700 1 -- 172.21.15.115:0/3493717982 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc01d070 con 0x7ff4e80fdee0 2023-07-14T18:10:33.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.541+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e819fbd0 con 0x7ff4e80fdee0 2023-07-14T18:10:33.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.541+0000 7ff4cffff700 1 -- 172.21.15.115:0/3493717982 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff4dc004cc0 con 0x7ff4e80fdee0 2023-07-14T18:10:33.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.541+0000 7ff4cffff700 1 -- 172.21.15.115:0/3493717982 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc005060 con 0x7ff4e80fdee0 2023-07-14T18:10:33.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.541+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e819fff0 con 0x7ff4e80fdee0 2023-07-14T18:10:33.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.541+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e810b2c0 con 0x7ff4e80fdee0 2023-07-14T18:10:33.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.542+0000 7ff4cffff700 1 -- 172.21.15.115:0/3493717982 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ff4dc003b70 con 0x7ff4e80fdee0 2023-07-14T18:10:33.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.543+0000 7ff4cffff700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4d00a1150 0x7ff4d00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.543+0000 7ff4e67fc700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4d00a1150 0x7ff4d00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.543+0000 7ff4cffff700 1 -- 172.21.15.115:0/3493717982 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff4dc0cdef0 con 0x7ff4e80fdee0 2023-07-14T18:10:33.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.543+0000 7ff4e67fc700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4d00a1150 0x7ff4d00a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8005950 tx=0x7ff4d80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:33.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.546+0000 7ff4cffff700 1 -- 172.21.15.115:0/3493717982 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff4dc097480 con 0x7ff4e80fdee0 2023-07-14T18:10:33.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.736+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff4e8106d30 con 0x7ff4e80fdee0 2023-07-14T18:10:33.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.736+0000 7ff4cffff700 1 -- 172.21.15.115:0/3493717982 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff4dc09aaa0 con 0x7ff4e80fdee0 2023-07-14T18:10:33.775 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 64 2023-07-14T18:10:33.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4d00a1150 msgr2=0x7ff4d00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4d00a1150 0x7ff4d00a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8005950 tx=0x7ff4d80058e0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 msgr2=0x7ff4e81008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 0x7ff4e81008c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc004730 tx=0x7ff4dc004760 comp rx=0 tx=0).stop 2023-07-14T18:10:33.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 shutdown_connections 2023-07-14T18:10:33.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4d00a1150 0x7ff4d00a3600 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4e81018e0 0x7ff4e80ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff4e80fdee0 0x7ff4e81008c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 --2- 172.21.15.115:0/3493717982 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff4e8100f50 0x7ff4e80fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 >> 172.21.15.115:0/3493717982 conn(0x7ff4e80f98a0 msgr2=0x7ff4e80fbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:33.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 shutdown_connections 2023-07-14T18:10:33.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.739+0000 7ff4ed4b7700 1 -- 172.21.15.115:0/3493717982 wait complete. 2023-07-14T18:10:33.789 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:10:33.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.875+0000 7f321f958700 1 -- 172.21.15.115:0/423560903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 msgr2=0x7f3218107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.875+0000 7f321f958700 1 --2- 172.21.15.115:0/423560903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 0x7f3218107010 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f321000b0a0 tx=0x7f321000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.875+0000 7f321f958700 1 -- 172.21.15.115:0/423560903 shutdown_connections 2023-07-14T18:10:33.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.875+0000 7f321f958700 1 --2- 172.21.15.115:0/423560903 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 0x7f3218107010 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.875+0000 7f321f958700 1 --2- 172.21.15.115:0/423560903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180feb30 0x7f32180fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.875+0000 7f321f958700 1 --2- 172.21.15.115:0/423560903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32180ffdc0 0x7f3218100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.875+0000 7f321f958700 1 -- 172.21.15.115:0/423560903 >> 172.21.15.115:0/423560903 conn(0x7f32180fa150 msgr2=0x7f32180fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:33.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.875+0000 7f321f958700 1 -- 172.21.15.115:0/423560903 shutdown_connections 2023-07-14T18:10:33.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.875+0000 7f321f958700 1 -- 172.21.15.115:0/423560903 wait complete. 2023-07-14T18:10:33.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321f958700 1 Processor -- start 2023-07-14T18:10:33.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321f958700 1 -- start start 2023-07-14T18:10:33.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321f958700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32180feb30 0x7f32181959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321f958700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180ffdc0 0x7f3218195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321f958700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 0x7f321819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321f958700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321819d570 con 0x7f3218100770 2023-07-14T18:10:33.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321f958700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321819d6b0 con 0x7f32180ffdc0 2023-07-14T18:10:33.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321f958700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321819d840 con 0x7f32180feb30 2023-07-14T18:10:33.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321def5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 0x7f321819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321def5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 0x7f321819cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52890/0 (socket says 172.21.15.115:52890) 2023-07-14T18:10:33.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321def5700 1 -- 172.21.15.115:0/3919004971 learned_addr learned my addr 172.21.15.115:0/3919004971 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:33.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321d6f4700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32180feb30 0x7f32181959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321cef3700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180ffdc0 0x7f3218195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321def5700 1 -- 172.21.15.115:0/3919004971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32180feb30 msgr2=0x7f32181959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321def5700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32180feb30 0x7f32181959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321def5700 1 -- 172.21.15.115:0/3919004971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180ffdc0 msgr2=0x7f3218195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:33.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321def5700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180ffdc0 0x7f3218195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:33.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321def5700 1 -- 172.21.15.115:0/3919004971 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210009d00 con 0x7f3218100770 2023-07-14T18:10:33.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f321def5700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 0x7f321819cfa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f32100062a0 tx=0x7f3210004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:33.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f320a7fc700 1 -- 172.21.15.115:0/3919004971 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321000e040 con 0x7f3218100770 2023-07-14T18:10:33.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.876+0000 7f320a7fc700 1 -- 172.21.15.115:0/3919004971 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f321001db20 con 0x7f3218100770 2023-07-14T18:10:33.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.877+0000 7f320a7fc700 1 -- 172.21.15.115:0/3919004971 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321001bac0 con 0x7f3218100770 2023-07-14T18:10:33.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.877+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321819dac0 con 0x7f3218100770 2023-07-14T18:10:33.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.877+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321819def0 con 0x7f3218100770 2023-07-14T18:10:33.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.878+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321810a1b0 con 0x7f3218100770 2023-07-14T18:10:33.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.879+0000 7f320a7fc700 1 -- 172.21.15.115:0/3919004971 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f321001dc90 con 0x7f3218100770 2023-07-14T18:10:33.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.879+0000 7f320a7fc700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f32040a1150 0x7f32040a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:33.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.879+0000 7f320a7fc700 1 -- 172.21.15.115:0/3919004971 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f32100ced60 con 0x7f3218100770 2023-07-14T18:10:33.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.879+0000 7f321d6f4700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f32040a1150 0x7f32040a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:33.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.880+0000 7f321d6f4700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f32040a1150 0x7f32040a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f320c005950 tx=0x7f320c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:33.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:33.882+0000 7f320a7fc700 1 -- 172.21.15.115:0/3919004971 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3210098350 con 0x7f3218100770 2023-07-14T18:10:34.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:33 smithi115 ceph-mon[127451]: pgmap v259: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 409 B/s wr, 2 op/s 2023-07-14T18:10:34.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:33 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2974835924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:10:34.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:33 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3560423453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:10:34.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:33 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3493717982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:10:34.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.072+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f321819e500 con 0x7f3218100770 2023-07-14T18:10:34.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.072+0000 7f320a7fc700 1 -- 172.21.15.115:0/3919004971 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3210021020 con 0x7f3218100770 2023-07-14T18:10:34.111 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:10:34.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f32040a1150 msgr2=0x7f32040a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f32040a1150 0x7f32040a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f320c005950 tx=0x7f320c00b410 comp rx=0 tx=0).stop 2023-07-14T18:10:34.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 msgr2=0x7f321819cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 0x7f321819cfa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f32100062a0 tx=0x7f3210004d00 comp rx=0 tx=0).stop 2023-07-14T18:10:34.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 shutdown_connections 2023-07-14T18:10:34.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f32040a1150 0x7f32040a3600 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3218100770 0x7f321819cfa0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32180feb30 0x7f32181959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 --2- 172.21.15.115:0/3919004971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180ffdc0 0x7f3218195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 >> 172.21.15.115:0/3919004971 conn(0x7f32180fa150 msgr2=0x7f32180689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:34.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 shutdown_connections 2023-07-14T18:10:34.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.075+0000 7f321f958700 1 -- 172.21.15.115:0/3919004971 wait complete. 2023-07-14T18:10:34.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T18:10:34.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.210+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3316176349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0105970 msgr2=0x7f87e0107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.210+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3316176349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0105970 0x7f87e0107d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f87d800b0a0 tx=0x7f87d800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.211+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3316176349 shutdown_connections 2023-07-14T18:10:34.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.211+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3316176349 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0105970 0x7f87e0107d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.211+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3316176349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87e0100730 0x7f87e0102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.211+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3316176349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87e0103050 0x7f87e0105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.211+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3316176349 >> 172.21.15.115:0/3316176349 conn(0x7f87e00fa130 msgr2=0x7f87e00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:34.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.211+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3316176349 shutdown_connections 2023-07-14T18:10:34.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.211+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3316176349 wait complete. 2023-07-14T18:10:34.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.211+0000 7f87e7f4f700 1 Processor -- start 2023-07-14T18:10:34.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.211+0000 7f87e7f4f700 1 -- start start 2023-07-14T18:10:34.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e7f4f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0100730 0x7f87e01000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:34.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e7f4f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87e0103050 0x7f87e00fe6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:34.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e7f4f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87e0105970 0x7f87e00fec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:34.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e7f4f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00ff3c0 con 0x7f87e0100730 2023-07-14T18:10:34.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e7f4f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00ff530 con 0x7f87e0105970 2023-07-14T18:10:34.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e7f4f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e019fa60 con 0x7f87e0103050 2023-07-14T18:10:34.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e5ceb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0100730 0x7f87e01000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:34.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e5ceb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0100730 0x7f87e01000a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52902/0 (socket says 172.21.15.115:52902) 2023-07-14T18:10:34.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e5ceb700 1 -- 172.21.15.115:0/3381062881 learned_addr learned my addr 172.21.15.115:0/3381062881 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:34.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e54ea700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87e0103050 0x7f87e00fe6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:34.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e64ec700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87e0105970 0x7f87e00fec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:34.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e5ceb700 1 -- 172.21.15.115:0/3381062881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87e0103050 msgr2=0x7f87e00fe6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e5ceb700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87e0103050 0x7f87e00fe6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e5ceb700 1 -- 172.21.15.115:0/3381062881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87e0105970 msgr2=0x7f87e00fec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e5ceb700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87e0105970 0x7f87e00fec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e5ceb700 1 -- 172.21.15.115:0/3381062881 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d8009d00 con 0x7f87e0100730 2023-07-14T18:10:34.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e64ec700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87e0105970 0x7f87e00fec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:34.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e5ceb700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0100730 0x7f87e01000a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00b530 tx=0x7f87dc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:34.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87d6ffd700 1 -- 172.21.15.115:0/3381062881 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc011830 con 0x7f87e0100730 2023-07-14T18:10:34.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87d6ffd700 1 -- 172.21.15.115:0/3381062881 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f87dc011e20 con 0x7f87e0100730 2023-07-14T18:10:34.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e019fd40 con 0x7f87e0100730 2023-07-14T18:10:34.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87d6ffd700 1 -- 172.21.15.115:0/3381062881 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc00f2e0 con 0x7f87e0100730 2023-07-14T18:10:34.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.212+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e01a01a0 con 0x7f87e0100730 2023-07-14T18:10:34.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.213+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e00fbca0 con 0x7f87e0100730 2023-07-14T18:10:34.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.215+0000 7f87d6ffd700 1 -- 172.21.15.115:0/3381062881 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f87dc011990 con 0x7f87e0100730 2023-07-14T18:10:34.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.216+0000 7f87d6ffd700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87cc0a1150 0x7f87cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:34.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.216+0000 7f87e54ea700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87cc0a1150 0x7f87cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:34.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.216+0000 7f87d6ffd700 1 -- 172.21.15.115:0/3381062881 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f87dc0cca50 con 0x7f87e0100730 2023-07-14T18:10:34.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.216+0000 7f87e54ea700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87cc0a1150 0x7f87cc0a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f87d0007c60 tx=0x7f87d00073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:34.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.219+0000 7f87d6ffd700 1 -- 172.21.15.115:0/3381062881 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f87dc095f50 con 0x7f87e0100730 2023-07-14T18:10:34.370 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:33 smithi150 ceph-mon[128853]: pgmap v259: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 409 B/s wr, 2 op/s 2023-07-14T18:10:34.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:33 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2974835924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:10:34.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:33 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3560423453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:10:34.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:33 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3493717982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:10:34.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:33 smithi187 ceph-mon[129317]: pgmap v259: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 409 B/s wr, 2 op/s 2023-07-14T18:10:34.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:33 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2974835924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:10:34.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:33 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3560423453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:10:34.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:33 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3493717982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:10:34.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.420+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f87e002c8e0 con 0x7f87e0100730 2023-07-14T18:10:34.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.420+0000 7f87d6ffd700 1 -- 172.21.15.115:0/3381062881 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+600 (secure 0 0 0) 0x7f87dc015270 con 0x7f87e0100730 2023-07-14T18:10:34.459 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6800/2169355495 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:34.459 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/4031581902 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:34.460 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/3311800921 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:34.460 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6801/2169355495 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:34.460 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2790161712 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:34.460 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2625519889 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:34.460 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6801/568655687 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:34.460 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/3340901626 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:34.460 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6800/568655687 2023-07-15T18:04:06.647341+0000 2023-07-14T18:10:34.461 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2416566340 2023-07-15T18:04:28.808488+0000 2023-07-14T18:10:34.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87cc0a1150 msgr2=0x7f87cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87cc0a1150 0x7f87cc0a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f87d0007c60 tx=0x7f87d00073d0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0100730 msgr2=0x7f87e01000a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0100730 0x7f87e01000a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00b530 tx=0x7f87dc00b840 comp rx=0 tx=0).stop 2023-07-14T18:10:34.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 shutdown_connections 2023-07-14T18:10:34.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87cc0a1150 0x7f87cc0a3600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87e0105970 0x7f87e00fec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87e0100730 0x7f87e01000a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 --2- 172.21.15.115:0/3381062881 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87e0103050 0x7f87e00fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 >> 172.21.15.115:0/3381062881 conn(0x7f87e00fa130 msgr2=0x7f87e0101390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:34.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 shutdown_connections 2023-07-14T18:10:34.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.423+0000 7f87e7f4f700 1 -- 172.21.15.115:0/3381062881 wait complete. 2023-07-14T18:10:34.463 INFO:teuthology.orchestra.run.smithi115.stderr:listed 10 entries 2023-07-14T18:10:34.473 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi115.front.sepia.ceph.com... 2023-07-14T18:10:34.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:34.473 DEBUG:teuthology.orchestra.run.smithi115:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:10:34.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi115.stderr:ceph-fuse[164740]: fuse finished with error 0 and tester_r 0 2023-07-14T18:10:34.509 INFO:teuthology.orchestra.run:waiting for 300 2023-07-14T18:10:34.545 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi115.front.sepia.ceph.com 2023-07-14T18:10:34.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:10:34.546 DEBUG:teuthology.orchestra.run.smithi115:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-07-14T18:10:34.620 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:10:34.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.743+0000 7fcf0c012700 1 -- 172.21.15.115:0/901233543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 msgr2=0x7fcf04107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.743+0000 7fcf0c012700 1 --2- 172.21.15.115:0/901233543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 0x7fcf04107d20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00b0a0 tx=0x7fcefc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.743+0000 7fcf0c012700 1 -- 172.21.15.115:0/901233543 shutdown_connections 2023-07-14T18:10:34.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.743+0000 7fcf0c012700 1 --2- 172.21.15.115:0/901233543 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 0x7fcf04107d20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.743+0000 7fcf0c012700 1 --2- 172.21.15.115:0/901233543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf041006e0 0x7fcf04102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.743+0000 7fcf0c012700 1 --2- 172.21.15.115:0/901233543 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04103000 0x7fcf041053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.743+0000 7fcf0c012700 1 -- 172.21.15.115:0/901233543 >> 172.21.15.115:0/901233543 conn(0x7fcf040fa100 msgr2=0x7fcf040fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:34.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.743+0000 7fcf0c012700 1 -- 172.21.15.115:0/901233543 shutdown_connections 2023-07-14T18:10:34.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.744+0000 7fcf0c012700 1 -- 172.21.15.115:0/901233543 wait complete. 2023-07-14T18:10:34.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.744+0000 7fcf0c012700 1 Processor -- start 2023-07-14T18:10:34.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.744+0000 7fcf0c012700 1 -- start start 2023-07-14T18:10:34.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.744+0000 7fcf0c012700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf041006e0 0x7fcf041958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:34.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.744+0000 7fcf0c012700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04103000 0x7fcf04195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:34.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0c012700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 0x7fcf0419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:34.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0c012700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0419d470 con 0x7fcf04105920 2023-07-14T18:10:34.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0c012700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0419d5b0 con 0x7fcf041006e0 2023-07-14T18:10:34.786 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0c012700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0419d740 con 0x7fcf04103000 2023-07-14T18:10:34.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0a5af700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 0x7fcf0419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:34.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0a5af700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 0x7fcf0419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52920/0 (socket says 172.21.15.115:52920) 2023-07-14T18:10:34.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0a5af700 1 -- 172.21.15.115:0/161991376 learned_addr learned my addr 172.21.15.115:0/161991376 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:34.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf095ad700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04103000 0x7fcf04195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:34.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf09dae700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf041006e0 0x7fcf041958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:34.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0a5af700 1 -- 172.21.15.115:0/161991376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04103000 msgr2=0x7fcf04195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.787 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0a5af700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04103000 0x7fcf04195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0a5af700 1 -- 172.21.15.115:0/161991376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf041006e0 msgr2=0x7fcf041958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0a5af700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf041006e0 0x7fcf041958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0a5af700 1 -- 172.21.15.115:0/161991376 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc009d00 con 0x7fcf04105920 2023-07-14T18:10:34.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf095ad700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04103000 0x7fcf04195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:34.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0a5af700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 0x7fcf0419cea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0062a0 tx=0x7fcefc004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:34.788 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcefaffd700 1 -- 172.21.15.115:0/161991376 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc00e040 con 0x7fcf04105920 2023-07-14T18:10:34.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcefaffd700 1 -- 172.21.15.115:0/161991376 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcefc0094c0 con 0x7fcf04105920 2023-07-14T18:10:34.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0419d9c0 con 0x7fcf04105920 2023-07-14T18:10:34.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcefaffd700 1 -- 172.21.15.115:0/161991376 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc01bc70 con 0x7fcf04105920 2023-07-14T18:10:34.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.745+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0419de90 con 0x7fcf04105920 2023-07-14T18:10:34.789 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.746+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf040fbc70 con 0x7fcf04105920 2023-07-14T18:10:34.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.747+0000 7fcefaffd700 1 -- 172.21.15.115:0/161991376 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fcefc01d890 con 0x7fcf04105920 2023-07-14T18:10:34.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.748+0000 7fcefaffd700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcef00a1220 0x7fcef00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:34.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.748+0000 7fcefaffd700 1 -- 172.21.15.115:0/161991376 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcefc0cf270 con 0x7fcf04105920 2023-07-14T18:10:34.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.748+0000 7fcf09dae700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcef00a1220 0x7fcef00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:34.790 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.748+0000 7fcf09dae700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcef00a1220 0x7fcef00a36d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf00005950 tx=0x7fcf000058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:34.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.751+0000 7fcefaffd700 1 -- 172.21.15.115:0/161991376 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcefc098800 con 0x7fcf04105920 2023-07-14T18:10:34.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.945+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf0402cb30 con 0x7fcf04105920 2023-07-14T18:10:34.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.946+0000 7fcefaffd700 1 -- 172.21.15.115:0/161991376 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7fcefc09be20 con 0x7fcf04105920 2023-07-14T18:10:34.985 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:34.985 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:34.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcef00a1220 msgr2=0x7fcef00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcef00a1220 0x7fcef00a36d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf00005950 tx=0x7fcf000058e0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 msgr2=0x7fcf0419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:34.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 0x7fcf0419cea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0062a0 tx=0x7fcefc004db0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 shutdown_connections 2023-07-14T18:10:34.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcef00a1220 0x7fcef00a36d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcf04105920 0x7fcf0419cea0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf041006e0 0x7fcf041958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 --2- 172.21.15.115:0/161991376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf04103000 0x7fcf04195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:34.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.948+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 >> 172.21.15.115:0/161991376 conn(0x7fcf040fa100 msgr2=0x7fcf04101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:34.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.949+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 shutdown_connections 2023-07-14T18:10:34.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:34.949+0000 7fcf0c012700 1 -- 172.21.15.115:0/161991376 wait complete. 2023-07-14T18:10:35.000 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:10:35.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.082+0000 7fb052169700 1 -- 172.21.15.115:0/2485299934 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0ffcf0 msgr2=0x7fb04c100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:35.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.082+0000 7fb052169700 1 --2- 172.21.15.115:0/2485299934 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0ffcf0 0x7fb04c100140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb040009990 tx=0x7fb040009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.082+0000 7fb052169700 1 -- 172.21.15.115:0/2485299934 shutdown_connections 2023-07-14T18:10:35.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.082+0000 7fb052169700 1 --2- 172.21.15.115:0/2485299934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb04c100680 0x7fb04c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.082+0000 7fb052169700 1 --2- 172.21.15.115:0/2485299934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c0feb00 0x7fb04c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.082+0000 7fb052169700 1 --2- 172.21.15.115:0/2485299934 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0ffcf0 0x7fb04c100140 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.082+0000 7fb052169700 1 -- 172.21.15.115:0/2485299934 >> 172.21.15.115:0/2485299934 conn(0x7fb04c0fa100 msgr2=0x7fb04c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:35.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.082+0000 7fb052169700 1 -- 172.21.15.115:0/2485299934 shutdown_connections 2023-07-14T18:10:35.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.082+0000 7fb052169700 1 -- 172.21.15.115:0/2485299934 wait complete. 2023-07-14T18:10:35.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.083+0000 7fb052169700 1 Processor -- start 2023-07-14T18:10:35.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.083+0000 7fb052169700 1 -- start start 2023-07-14T18:10:35.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.083+0000 7fb052169700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0feb00 0x7fb04c068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:35.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.083+0000 7fb052169700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb04c0ffcf0 0x7fb04c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:35.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.083+0000 7fb052169700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c100680 0x7fb04c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:35.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.083+0000 7fb052169700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c19d4c0 con 0x7fb04c0feb00 2023-07-14T18:10:35.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.083+0000 7fb052169700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c19d630 con 0x7fb04c100680 2023-07-14T18:10:35.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.083+0000 7fb052169700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c19d7f0 con 0x7fb04c0ffcf0 2023-07-14T18:10:35.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.083+0000 7fb04b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0feb00 0x7fb04c068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:35.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb04b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0feb00 0x7fb04c068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52930/0 (socket says 172.21.15.115:52930) 2023-07-14T18:10:35.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb04b7fe700 1 -- 172.21.15.115:0/770825355 learned_addr learned my addr 172.21.15.115:0/770825355 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:35.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb04affd700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb04c0ffcf0 0x7fb04c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:35.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb04b7fe700 1 -- 172.21.15.115:0/770825355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb04c0ffcf0 msgr2=0x7fb04c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:35.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb04b7fe700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb04c0ffcf0 0x7fb04c19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb04b7fe700 1 -- 172.21.15.115:0/770825355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c100680 msgr2=0x7fb04c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:35.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb04b7fe700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c100680 0x7fb04c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb04b7fe700 1 -- 172.21.15.115:0/770825355 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0400095f0 con 0x7fb04c0feb00 2023-07-14T18:10:35.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb04b7fe700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0feb00 0x7fb04c068310 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00b500 tx=0x7fb03c00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:35.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb048ff9700 1 -- 172.21.15.115:0/770825355 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c011830 con 0x7fb04c0feb00 2023-07-14T18:10:35.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb048ff9700 1 -- 172.21.15.115:0/770825355 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb03c011e20 con 0x7fb04c0feb00 2023-07-14T18:10:35.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c19dad0 con 0x7fb04c0feb00 2023-07-14T18:10:35.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb048ff9700 1 -- 172.21.15.115:0/770825355 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c010dd0 con 0x7fb04c0feb00 2023-07-14T18:10:35.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.084+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c19e0b0 con 0x7fb04c0feb00 2023-07-14T18:10:35.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.085+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04c10a2a0 con 0x7fb04c0feb00 2023-07-14T18:10:35.128 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.086+0000 7fb048ff9700 1 -- 172.21.15.115:0/770825355 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fb03c011990 con 0x7fb04c0feb00 2023-07-14T18:10:35.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.086+0000 7fb048ff9700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb0340a1220 0x7fb0340a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:35.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.087+0000 7fb048ff9700 1 -- 172.21.15.115:0/770825355 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb03c0cc600 con 0x7fb04c0feb00 2023-07-14T18:10:35.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.087+0000 7fb04affd700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb0340a1220 0x7fb0340a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:35.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.087+0000 7fb04affd700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb0340a1220 0x7fb0340a36d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c010 tx=0x7fb04000b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:35.129 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.091+0000 7fb048ff9700 1 -- 172.21.15.115:0/770825355 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb03c095b00 con 0x7fb04c0feb00 2023-07-14T18:10:35.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.280+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb04c105a80 con 0x7fb04c0feb00 2023-07-14T18:10:35.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.282+0000 7fb048ff9700 1 -- 172.21.15.115:0/770825355 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7fb03c099120 con 0x7fb04c0feb00 2023-07-14T18:10:35.320 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:35.321 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:35.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb0340a1220 msgr2=0x7fb0340a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:35.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb0340a1220 0x7fb0340a36d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c010 tx=0x7fb04000b540 comp rx=0 tx=0).stop 2023-07-14T18:10:35.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0feb00 msgr2=0x7fb04c068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:35.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0feb00 0x7fb04c068310 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00b500 tx=0x7fb03c00b8c0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 shutdown_connections 2023-07-14T18:10:35.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb0340a1220 0x7fb0340a36d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb04c100680 0x7fb04c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb04c0feb00 0x7fb04c068310 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 --2- 172.21.15.115:0/770825355 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb04c0ffcf0 0x7fb04c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 >> 172.21.15.115:0/770825355 conn(0x7fb04c0fa100 msgr2=0x7fb04c0fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:35.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 shutdown_connections 2023-07-14T18:10:35.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.284+0000 7fb052169700 1 -- 172.21.15.115:0/770825355 wait complete. 2023-07-14T18:10:35.333 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:34 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3919004971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:10:35.333 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:34 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3381062881' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:10:35.334 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:34 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/161991376' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:35.335 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-14T18:10:35.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:34 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3919004971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:10:35.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:34 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3381062881' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:10:35.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:34 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/161991376' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:35.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:34 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3919004971' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:10:35.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:34 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3381062881' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:10:35.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:34 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/161991376' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:35.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 -- 172.21.15.115:0/4043119455 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce201006e0 msgr2=0x7fce20106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:35.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 --2- 172.21.15.115:0/4043119455 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce201006e0 0x7fce20106f80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fce1800b0a0 tx=0x7fce1800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 -- 172.21.15.115:0/4043119455 shutdown_connections 2023-07-14T18:10:35.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 --2- 172.21.15.115:0/4043119455 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce201006e0 0x7fce20106f80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 --2- 172.21.15.115:0/4043119455 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce200ff020 0x7fce200ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 --2- 172.21.15.115:0/4043119455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce200ffd30 0x7fce201001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 -- 172.21.15.115:0/4043119455 >> 172.21.15.115:0/4043119455 conn(0x7fce200f98a0 msgr2=0x7fce200fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:35.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 -- 172.21.15.115:0/4043119455 shutdown_connections 2023-07-14T18:10:35.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 -- 172.21.15.115:0/4043119455 wait complete. 2023-07-14T18:10:35.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.420+0000 7fce26bcf700 1 Processor -- start 2023-07-14T18:10:35.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce26bcf700 1 -- start start 2023-07-14T18:10:35.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce26bcf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce200ff020 0x7fce201948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:35.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce26bcf700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce200ffd30 0x7fce20194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:35.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce26bcf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce201006e0 0x7fce2019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:35.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce26bcf700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2019c430 con 0x7fce200ffd30 2023-07-14T18:10:35.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce26bcf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2019c570 con 0x7fce201006e0 2023-07-14T18:10:35.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce26bcf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2019c700 con 0x7fce200ff020 2023-07-14T18:10:35.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce1ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce200ffd30 0x7fce20194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:35.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce1ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce200ffd30 0x7fce20194df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52946/0 (socket says 172.21.15.115:52946) 2023-07-14T18:10:35.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce1ffff700 1 -- 172.21.15.115:0/3104962552 learned_addr learned my addr 172.21.15.115:0/3104962552 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:35.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce1ffff700 1 -- 172.21.15.115:0/3104962552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce200ff020 msgr2=0x7fce201948b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:35.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce2496b700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce200ff020 0x7fce201948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:35.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce2516c700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce201006e0 0x7fce2019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:35.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce1ffff700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce200ff020 0x7fce201948b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce1ffff700 1 -- 172.21.15.115:0/3104962552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce201006e0 msgr2=0x7fce2019be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:35.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce1ffff700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce201006e0 0x7fce2019be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:35.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce1ffff700 1 -- 172.21.15.115:0/3104962552 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce18009d00 con 0x7fce200ffd30 2023-07-14T18:10:35.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.421+0000 7fce1ffff700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce200ffd30 0x7fce20194df0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00d980 tx=0x7fce0c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:35.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.422+0000 7fce1dffb700 1 -- 172.21.15.115:0/3104962552 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c00de60 con 0x7fce200ffd30 2023-07-14T18:10:35.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.422+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2019c9e0 con 0x7fce200ffd30 2023-07-14T18:10:35.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.422+0000 7fce1dffb700 1 -- 172.21.15.115:0/3104962552 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fce0c0044b0 con 0x7fce200ffd30 2023-07-14T18:10:35.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.422+0000 7fce1dffb700 1 -- 172.21.15.115:0/3104962552 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c003b40 con 0x7fce200ffd30 2023-07-14T18:10:35.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.422+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2019cf40 con 0x7fce200ffd30 2023-07-14T18:10:35.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.423+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2010a190 con 0x7fce200ffd30 2023-07-14T18:10:35.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.424+0000 7fce1dffb700 1 -- 172.21.15.115:0/3104962552 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fce0c003ca0 con 0x7fce200ffd30 2023-07-14T18:10:35.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.424+0000 7fce1dffb700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fce100a1150 0x7fce100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:35.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.424+0000 7fce2496b700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fce100a1150 0x7fce100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:35.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.424+0000 7fce1dffb700 1 -- 172.21.15.115:0/3104962552 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fce0c0cc830 con 0x7fce200ffd30 2023-07-14T18:10:35.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.425+0000 7fce2496b700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fce100a1150 0x7fce100a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fce20067310 tx=0x7fce14005cf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:35.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.428+0000 7fce1dffb700 1 -- 172.21.15.115:0/3104962552 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce0c095d90 con 0x7fce200ffd30 2023-07-14T18:10:35.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.674+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fce2010a350 con 0x7fce200ffd30 2023-07-14T18:10:36.000 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.961+0000 7fce1dffb700 1 -- 172.21.15.115:0/3104962552 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fce2010a350 con 0x7fce200ffd30 2023-07-14T18:10:36.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fce100a1150 msgr2=0x7fce100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:36.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fce100a1150 0x7fce100a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fce20067310 tx=0x7fce14005cf0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce200ffd30 msgr2=0x7fce20194df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:36.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce200ffd30 0x7fce20194df0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00d980 tx=0x7fce0c00dd40 comp rx=0 tx=0).stop 2023-07-14T18:10:36.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 shutdown_connections 2023-07-14T18:10:36.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fce100a1150 0x7fce100a3600 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce201006e0 0x7fce2019be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce200ff020 0x7fce201948b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 --2- 172.21.15.115:0/3104962552 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fce200ffd30 0x7fce20194df0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 >> 172.21.15.115:0/3104962552 conn(0x7fce200f98a0 msgr2=0x7fce200fbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:36.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 shutdown_connections 2023-07-14T18:10:36.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:35.964+0000 7fce26bcf700 1 -- 172.21.15.115:0/3104962552 wait complete. 2023-07-14T18:10:36.015 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-07-14T18:10:36.016 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-07-14T18:10:36.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.101+0000 7f42317c1700 1 -- 172.21.15.115:0/3809999518 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c1000f0 msgr2=0x7f422c100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:36.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.101+0000 7f42317c1700 1 --2- 172.21.15.115:0/3809999518 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c1000f0 0x7f422c100500 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f421c009990 tx=0x7f421c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.101+0000 7f42317c1700 1 -- 172.21.15.115:0/3809999518 shutdown_connections 2023-07-14T18:10:36.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.101+0000 7f42317c1700 1 --2- 172.21.15.115:0/3809999518 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c0ff6f0 0x7f422c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.101+0000 7f42317c1700 1 --2- 172.21.15.115:0/3809999518 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c1000f0 0x7f422c100500 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.101+0000 7f42317c1700 1 --2- 172.21.15.115:0/3809999518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f422c0fed40 0x7f422c0ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.101+0000 7f42317c1700 1 -- 172.21.15.115:0/3809999518 >> 172.21.15.115:0/3809999518 conn(0x7f422c0f98e0 msgr2=0x7f422c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:36.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.101+0000 7f42317c1700 1 -- 172.21.15.115:0/3809999518 shutdown_connections 2023-07-14T18:10:36.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.101+0000 7f42317c1700 1 -- 172.21.15.115:0/3809999518 wait complete. 2023-07-14T18:10:36.145 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.102+0000 7f42317c1700 1 Processor -- start 2023-07-14T18:10:36.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.102+0000 7f42317c1700 1 -- start start 2023-07-14T18:10:36.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f42317c1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c0fed40 0x7f422c0fe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:36.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f42317c1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f422c0ff6f0 0x7f422c19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:36.146 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f42317c1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1000f0 0x7f422c19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:36.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f42317c1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c19d440 con 0x7f422c0fed40 2023-07-14T18:10:36.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f42317c1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c19d5b0 con 0x7f422c0ff6f0 2023-07-14T18:10:36.147 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f42317c1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c19d770 con 0x7f422c1000f0 2023-07-14T18:10:36.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c0fed40 0x7f422c0fe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:36.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c0fed40 0x7f422c0fe7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52958/0 (socket says 172.21.15.115:52958) 2023-07-14T18:10:36.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422affd700 1 -- 172.21.15.115:0/202927191 learned_addr learned my addr 172.21.15.115:0/202927191 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:36.148 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422a7fc700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f422c0ff6f0 0x7f422c19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:36.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422affd700 1 -- 172.21.15.115:0/202927191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1000f0 msgr2=0x7f422c19cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:36.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422affd700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1000f0 0x7f422c19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422affd700 1 -- 172.21.15.115:0/202927191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f422c0ff6f0 msgr2=0x7f422c19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:36.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422affd700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f422c0ff6f0 0x7f422c19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:36.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422affd700 1 -- 172.21.15.115:0/202927191 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c0095f0 con 0x7f422c0fed40 2023-07-14T18:10:36.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f422affd700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c0fed40 0x7f422c0fe7c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f421c006010 tx=0x7f421c003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:36.149 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f4213fff700 1 -- 172.21.15.115:0/202927191 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c01d070 con 0x7f422c0fed40 2023-07-14T18:10:36.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f4213fff700 1 -- 172.21.15.115:0/202927191 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f421c004640 con 0x7f422c0fed40 2023-07-14T18:10:36.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c19d9f0 con 0x7f422c0fed40 2023-07-14T18:10:36.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f4213fff700 1 -- 172.21.15.115:0/202927191 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c00f460 con 0x7f422c0fed40 2023-07-14T18:10:36.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.103+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c19de10 con 0x7f422c0fed40 2023-07-14T18:10:36.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.104+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c106140 con 0x7f422c0fed40 2023-07-14T18:10:36.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.105+0000 7f4213fff700 1 -- 172.21.15.115:0/202927191 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f421c022470 con 0x7f422c0fed40 2023-07-14T18:10:36.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.106+0000 7f4213fff700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42140a1150 0x7f42140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:36.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.106+0000 7f422a7fc700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42140a1150 0x7f42140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:36.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.106+0000 7f4213fff700 1 -- 172.21.15.115:0/202927191 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f421c0ce1e0 con 0x7f422c0fed40 2023-07-14T18:10:36.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.106+0000 7f422a7fc700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42140a1150 0x7f42140a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4220005fd0 tx=0x7f422000a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:36.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.109+0000 7f4213fff700 1 -- 172.21.15.115:0/202927191 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f421c097870 con 0x7f422c0fed40 2023-07-14T18:10:36.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:35 smithi115 ceph-mon[127451]: pgmap v260: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 268 B/s wr, 0 op/s 2023-07-14T18:10:36.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:35 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/770825355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:36.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:35 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3104962552' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T18:10:36.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:35 smithi150 ceph-mon[128853]: pgmap v260: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 268 B/s wr, 0 op/s 2023-07-14T18:10:36.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:35 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/770825355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:36.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:35 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3104962552' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T18:10:36.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.341+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f422c10a220 con 0x7f422c0fed40 2023-07-14T18:10:36.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:35 smithi187 ceph-mon[129317]: pgmap v260: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 268 B/s wr, 0 op/s 2023-07-14T18:10:36.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:35 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/770825355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:36.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:35 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3104962552' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T18:10:37.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.968+0000 7f4213fff700 1 -- 172.21.15.115:0/202927191 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f421c09ae90 con 0x7f422c0fed40 2023-07-14T18:10:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42140a1150 msgr2=0x7f42140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42140a1150 0x7f42140a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4220005fd0 tx=0x7f422000a560 comp rx=0 tx=0).stop 2023-07-14T18:10:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c0fed40 msgr2=0x7f422c0fe7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c0fed40 0x7f422c0fe7c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f421c006010 tx=0x7f421c003b00 comp rx=0 tx=0).stop 2023-07-14T18:10:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 shutdown_connections 2023-07-14T18:10:37.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42140a1150 0x7f42140a3600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f422c1000f0 0x7f422c19cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f422c0fed40 0x7f422c0fe7c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 --2- 172.21.15.115:0/202927191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f422c0ff6f0 0x7f422c19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 >> 172.21.15.115:0/202927191 conn(0x7f422c0f98e0 msgr2=0x7f422c103280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 shutdown_connections 2023-07-14T18:10:37.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:36.970+0000 7f42317c1700 1 -- 172.21.15.115:0/202927191 wait complete. 2023-07-14T18:10:37.024 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-14T18:10:37.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 -- 172.21.15.115:0/3633546705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ff020 msgr2=0x7fe8080ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 --2- 172.21.15.115:0/3633546705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ff020 0x7fe8080ff430 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009990 tx=0x7fe7fc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 -- 172.21.15.115:0/3633546705 shutdown_connections 2023-07-14T18:10:37.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 --2- 172.21.15.115:0/3633546705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8081006e0 0x7fe808106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.150 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 --2- 172.21.15.115:0/3633546705 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ff020 0x7fe8080ff430 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 --2- 172.21.15.115:0/3633546705 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8080ffd30 0x7fe8081001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 -- 172.21.15.115:0/3633546705 >> 172.21.15.115:0/3633546705 conn(0x7fe8080f98a0 msgr2=0x7fe8080fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:37.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 -- 172.21.15.115:0/3633546705 shutdown_connections 2023-07-14T18:10:37.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 -- 172.21.15.115:0/3633546705 wait complete. 2023-07-14T18:10:37.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 Processor -- start 2023-07-14T18:10:37.151 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.111+0000 7fe80efee700 1 -- start start 2023-07-14T18:10:37.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80efee700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8080ff020 0x7fe808194790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:37.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80efee700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ffd30 0x7fe808194cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:37.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80efee700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8081006e0 0x7fe80819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:37.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80efee700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80819c310 con 0x7fe8080ffd30 2023-07-14T18:10:37.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80efee700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80819c450 con 0x7fe8080ff020 2023-07-14T18:10:37.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80efee700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80819c5e0 con 0x7fe8081006e0 2023-07-14T18:10:37.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe807fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ffd30 0x7fe808194cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:37.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe807fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ffd30 0x7fe808194cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52974/0 (socket says 172.21.15.115:52974) 2023-07-14T18:10:37.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe807fff700 1 -- 172.21.15.115:0/2595436753 learned_addr learned my addr 172.21.15.115:0/2595436753 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:37.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80d58b700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8081006e0 0x7fe80819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:37.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80cd8a700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8080ff020 0x7fe808194790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:37.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe807fff700 1 -- 172.21.15.115:0/2595436753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8081006e0 msgr2=0x7fe80819bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe807fff700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8081006e0 0x7fe80819bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe807fff700 1 -- 172.21.15.115:0/2595436753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8080ff020 msgr2=0x7fe808194790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe807fff700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8080ff020 0x7fe808194790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe807fff700 1 -- 172.21.15.115:0/2595436753 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0095f0 con 0x7fe8080ffd30 2023-07-14T18:10:37.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80d58b700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8081006e0 0x7fe80819bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:37.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe807fff700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ffd30 0x7fe808194cd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400c990 tx=0x7fe7f400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:37.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe805ffb700 1 -- 172.21.15.115:0/2595436753 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4007b00 con 0x7fe8080ffd30 2023-07-14T18:10:37.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe805ffb700 1 -- 172.21.15.115:0/2595436753 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe7f4007c60 con 0x7fe8080ffd30 2023-07-14T18:10:37.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80819c8c0 con 0x7fe8080ffd30 2023-07-14T18:10:37.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.112+0000 7fe805ffb700 1 -- 172.21.15.115:0/2595436753 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4018660 con 0x7fe8080ffd30 2023-07-14T18:10:37.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.113+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80819ce20 con 0x7fe8080ffd30 2023-07-14T18:10:37.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.113+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe808105a70 con 0x7fe8080ffd30 2023-07-14T18:10:37.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.114+0000 7fe805ffb700 1 -- 172.21.15.115:0/2595436753 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fe7f400f450 con 0x7fe8080ffd30 2023-07-14T18:10:37.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.115+0000 7fe805ffb700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a11d0 0x7fe7f80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:37.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.115+0000 7fe805ffb700 1 -- 172.21.15.115:0/2595436753 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe7f40ccb30 con 0x7fe8080ffd30 2023-07-14T18:10:37.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.115+0000 7fe80cd8a700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a11d0 0x7fe7f80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:37.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.115+0000 7fe80cd8a700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a11d0 0x7fe7f80a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00b5c0 tx=0x7fe7fc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:37.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.119+0000 7fe805ffb700 1 -- 172.21.15.115:0/2595436753 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe7f4096030 con 0x7fe8080ffd30 2023-07-14T18:10:37.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:36 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3104962552' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T18:10:37.355 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:36 smithi115 ceph-mon[127451]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:37.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:36 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/202927191' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T18:10:37.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:36 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3104962552' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T18:10:37.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:36 smithi150 ceph-mon[128853]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:37.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:36 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/202927191' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T18:10:37.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.355+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fe80802c8e0 con 0x7fe8080ffd30 2023-07-14T18:10:37.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:36 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3104962552' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T18:10:37.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:36 smithi187 ceph-mon[129317]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:37.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:36 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/202927191' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T18:10:37.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.414+0000 7fe805ffb700 1 -- 172.21.15.115:0/2595436753 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7fe7f4099650 con 0x7fe8080ffd30 2023-07-14T18:10:37.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a11d0 msgr2=0x7fe7f80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a11d0 0x7fe7f80a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00b5c0 tx=0x7fe7fc005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ffd30 msgr2=0x7fe808194cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ffd30 0x7fe808194cd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400c990 tx=0x7fe7f400cd50 comp rx=0 tx=0).stop 2023-07-14T18:10:37.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 shutdown_connections 2023-07-14T18:10:37.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a11d0 0x7fe7f80a3680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8081006e0 0x7fe80819bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8080ff020 0x7fe808194790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 --2- 172.21.15.115:0/2595436753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8080ffd30 0x7fe808194cd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 >> 172.21.15.115:0/2595436753 conn(0x7fe8080f98a0 msgr2=0x7fe8080fbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:37.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 shutdown_connections 2023-07-14T18:10:37.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.417+0000 7fe80efee700 1 -- 172.21.15.115:0/2595436753 wait complete. 2023-07-14T18:10:37.471 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-14T18:10:37.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.556+0000 7f3df4ec1700 1 -- 172.21.15.115:0/2890092654 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df00ff6f0 msgr2=0x7f3df0106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.556+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/2890092654 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df00ff6f0 0x7f3df0106fa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3de800b0a0 tx=0x7f3de800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 -- 172.21.15.115:0/2890092654 shutdown_connections 2023-07-14T18:10:37.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/2890092654 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df00ff6f0 0x7f3df0106fa0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/2890092654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df0100180 0x7f3df0100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/2890092654 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df00fed40 0x7f3df00ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 -- 172.21.15.115:0/2890092654 >> 172.21.15.115:0/2890092654 conn(0x7f3df00f98a0 msgr2=0x7f3df00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:37.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 -- 172.21.15.115:0/2890092654 shutdown_connections 2023-07-14T18:10:37.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 -- 172.21.15.115:0/2890092654 wait complete. 2023-07-14T18:10:37.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 Processor -- start 2023-07-14T18:10:37.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 -- start start 2023-07-14T18:10:37.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.557+0000 7f3df4ec1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df00fed40 0x7f3df00fe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:37.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3df4ec1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df00ff6f0 0x7f3df019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:37.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3df4ec1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df0100180 0x7f3df019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:37.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3df4ec1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df019d500 con 0x7f3df0100180 2023-07-14T18:10:37.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3df4ec1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df019d670 con 0x7f3df00ff6f0 2023-07-14T18:10:37.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3df4ec1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df019d830 con 0x7f3df00fed40 2023-07-14T18:10:37.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3deed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df0100180 0x7f3df019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:37.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3deed9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df0100180 0x7f3df019cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:52990/0 (socket says 172.21.15.115:52990) 2023-07-14T18:10:37.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3deed9d700 1 -- 172.21.15.115:0/903433874 learned_addr learned my addr 172.21.15.115:0/903433874 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:37.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3dedd9b700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df00ff6f0 0x7f3df019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:37.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3dee59c700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df00fed40 0x7f3df00fe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:37.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3deed9d700 1 -- 172.21.15.115:0/903433874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df00fed40 msgr2=0x7f3df00fe7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3deed9d700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df00fed40 0x7f3df00fe7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3deed9d700 1 -- 172.21.15.115:0/903433874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df00ff6f0 msgr2=0x7f3df019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:37.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3deed9d700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df00ff6f0 0x7f3df019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:37.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3deed9d700 1 -- 172.21.15.115:0/903433874 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8009d00 con 0x7f3df0100180 2023-07-14T18:10:37.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3deed9d700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df0100180 0x7f3df019cd30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3de800d010 tx=0x7f3de800bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:37.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3ddf7fe700 1 -- 172.21.15.115:0/903433874 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de800e040 con 0x7f3df0100180 2023-07-14T18:10:37.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df019dab0 con 0x7f3df0100180 2023-07-14T18:10:37.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3ddf7fe700 1 -- 172.21.15.115:0/903433874 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3de8003e10 con 0x7f3df0100180 2023-07-14T18:10:37.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3ddf7fe700 1 -- 172.21.15.115:0/903433874 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8005260 con 0x7f3df0100180 2023-07-14T18:10:37.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.558+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df019df50 con 0x7f3df0100180 2023-07-14T18:10:37.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.559+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df0105ae0 con 0x7f3df0100180 2023-07-14T18:10:37.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.560+0000 7f3ddf7fe700 1 -- 172.21.15.115:0/903433874 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f3de801da10 con 0x7f3df0100180 2023-07-14T18:10:37.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.561+0000 7f3ddf7fe700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f3dd80a1150 0x7f3dd80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:37.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.561+0000 7f3dee59c700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f3dd80a1150 0x7f3dd80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:37.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.561+0000 7f3ddf7fe700 1 -- 172.21.15.115:0/903433874 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3de80ce4f0 con 0x7f3df0100180 2023-07-14T18:10:37.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.562+0000 7f3dee59c700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f3dd80a1150 0x7f3dd80a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009bf0 tx=0x7f3de0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:37.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.565+0000 7f3ddf7fe700 1 -- 172.21.15.115:0/903433874 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3de8097c00 con 0x7f3df0100180 2023-07-14T18:10:37.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:37.798+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f3df002cd00 con 0x7f3df0100180 2023-07-14T18:10:38.266 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: pgmap v261: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-14T18:10:38.266 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T18:10:38.266 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/202927191' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T18:10:38.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T18:10:38.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:38.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T18:10:38.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T18:10:38.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2595436753' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T18:10:38.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] up:active 2023-07-14T18:10:38.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2595436753' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T18:10:38.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T18:10:38.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T18:10:38.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: Cluster is now healthy 2023-07-14T18:10:38.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2023-07-14T18:10:38.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2023-07-14T18:10:38.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: daemon mds.f is now active in filesystem cephfs as rank 2 2023-07-14T18:10:38.268 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:37 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/903433874' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T18:10:38.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: pgmap v261: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-14T18:10:38.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T18:10:38.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/202927191' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T18:10:38.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T18:10:38.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:38.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T18:10:38.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T18:10:38.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2595436753' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T18:10:38.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] up:active 2023-07-14T18:10:38.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2595436753' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T18:10:38.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T18:10:38.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T18:10:38.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: Cluster is now healthy 2023-07-14T18:10:38.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2023-07-14T18:10:38.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2023-07-14T18:10:38.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: daemon mds.f is now active in filesystem cephfs as rank 2 2023-07-14T18:10:38.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:37 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/903433874' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T18:10:38.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: pgmap v261: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-14T18:10:38.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T18:10:38.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/202927191' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T18:10:38.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T18:10:38.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:10:38.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T18:10:38.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T18:10:38.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2595436753' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T18:10:38.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] up:active 2023-07-14T18:10:38.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2595436753' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T18:10:38.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T18:10:38.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T18:10:38.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: Cluster is now healthy 2023-07-14T18:10:38.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2023-07-14T18:10:38.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2023-07-14T18:10:38.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: daemon mds.f is now active in filesystem cephfs as rank 2 2023-07-14T18:10:38.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:37 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/903433874' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T18:10:38.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.421+0000 7f3ddf7fe700 1 -- 172.21.15.115:0/903433874 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3de809b220 con 0x7f3df0100180 2023-07-14T18:10:38.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f3dd80a1150 msgr2=0x7f3dd80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:38.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f3dd80a1150 0x7f3dd80a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3de0009bf0 tx=0x7f3de0009260 comp rx=0 tx=0).stop 2023-07-14T18:10:38.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df0100180 msgr2=0x7f3df019cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:38.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df0100180 0x7f3df019cd30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3de800d010 tx=0x7f3de800bcc0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 shutdown_connections 2023-07-14T18:10:38.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f3dd80a1150 0x7f3dd80a3600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3df0100180 0x7f3df019cd30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3df00fed40 0x7f3df00fe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 --2- 172.21.15.115:0/903433874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df00ff6f0 0x7f3df019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.424+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 >> 172.21.15.115:0/903433874 conn(0x7f3df00f98a0 msgr2=0x7f3df00fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:38.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.425+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 shutdown_connections 2023-07-14T18:10:38.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.425+0000 7f3df4ec1700 1 -- 172.21.15.115:0/903433874 wait complete. 2023-07-14T18:10:38.477 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2023-07-14T18:10:38.478 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-07-14T18:10:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.569+0000 7ff7d8959700 1 -- 172.21.15.115:0/3842748668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 msgr2=0x7ff7d4101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.569+0000 7ff7d8959700 1 --2- 172.21.15.115:0/3842748668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 0x7ff7d4101310 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009990 tx=0x7ff7c8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.569+0000 7ff7d8959700 1 -- 172.21.15.115:0/3842748668 shutdown_connections 2023-07-14T18:10:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.569+0000 7ff7d8959700 1 --2- 172.21.15.115:0/3842748668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d4101850 0x7ff7d41080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.569+0000 7ff7d8959700 1 --2- 172.21.15.115:0/3842748668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7d40ffc10 0x7ff7d4100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.569+0000 7ff7d8959700 1 --2- 172.21.15.115:0/3842748668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 0x7ff7d4101310 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 -- 172.21.15.115:0/3842748668 >> 172.21.15.115:0/3842748668 conn(0x7ff7d40fb210 msgr2=0x7ff7d40fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 -- 172.21.15.115:0/3842748668 shutdown_connections 2023-07-14T18:10:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 -- 172.21.15.115:0/3842748668 wait complete. 2023-07-14T18:10:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 Processor -- start 2023-07-14T18:10:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 -- start start 2023-07-14T18:10:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7d40ffc10 0x7ff7d4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 0x7ff7d4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d4101850 0x7ff7d419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d419f670 con 0x7ff7d4100ea0 2023-07-14T18:10:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d419f7b0 con 0x7ff7d4101850 2023-07-14T18:10:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.570+0000 7ff7d8959700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d419f940 con 0x7ff7d40ffc10 2023-07-14T18:10:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d1d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 0x7ff7d4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d1d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 0x7ff7d4198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53002/0 (socket says 172.21.15.115:53002) 2023-07-14T18:10:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d1d9b700 1 -- 172.21.15.115:0/235015282 learned_addr learned my addr 172.21.15.115:0/235015282 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d259c700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7d40ffc10 0x7ff7d4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d2d9d700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d4101850 0x7ff7d419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d1d9b700 1 -- 172.21.15.115:0/235015282 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7d40ffc10 msgr2=0x7ff7d4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d1d9b700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7d40ffc10 0x7ff7d4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d1d9b700 1 -- 172.21.15.115:0/235015282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d4101850 msgr2=0x7ff7d419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d1d9b700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d4101850 0x7ff7d419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.613 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d1d9b700 1 -- 172.21.15.115:0/235015282 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c80095f0 con 0x7ff7d4100ea0 2023-07-14T18:10:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d1d9b700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 0x7ff7d4198030 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8006010 tx=0x7ff7c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7c37fe700 1 -- 172.21.15.115:0/235015282 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c801d070 con 0x7ff7d4100ea0 2023-07-14T18:10:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7c37fe700 1 -- 172.21.15.115:0/235015282 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff7c8004000 con 0x7ff7d4100ea0 2023-07-14T18:10:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d419fbc0 con 0x7ff7d4100ea0 2023-07-14T18:10:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7c37fe700 1 -- 172.21.15.115:0/235015282 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8022b90 con 0x7ff7d4100ea0 2023-07-14T18:10:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.571+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d41a0060 con 0x7ff7d4100ea0 2023-07-14T18:10:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.572+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d41921d0 con 0x7ff7d4100ea0 2023-07-14T18:10:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.573+0000 7ff7c37fe700 1 -- 172.21.15.115:0/235015282 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ff7c8003b70 con 0x7ff7d4100ea0 2023-07-14T18:10:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.574+0000 7ff7c37fe700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff7bc0a1150 0x7ff7bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:38.615 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.574+0000 7ff7d259c700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff7bc0a1150 0x7ff7bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.574+0000 7ff7c37fe700 1 -- 172.21.15.115:0/235015282 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff7c80cdd90 con 0x7ff7d4100ea0 2023-07-14T18:10:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.574+0000 7ff7d259c700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff7bc0a1150 0x7ff7bc0a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009a10 tx=0x7ff7c4009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:38.616 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.578+0000 7ff7c37fe700 1 -- 172.21.15.115:0/235015282 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff7c805c780 con 0x7ff7d4100ea0 2023-07-14T18:10:38.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.802+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7d4060a80 con 0x7ff7bc0a1150 2023-07-14T18:10:38.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.848+0000 7ff7c37fe700 1 -- 172.21.15.115:0/235015282 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff7d4060a80 con 0x7ff7bc0a1150 2023-07-14T18:10:38.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff7bc0a1150 msgr2=0x7ff7bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:38.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff7bc0a1150 0x7ff7bc0a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009a10 tx=0x7ff7c4009190 comp rx=0 tx=0).stop 2023-07-14T18:10:38.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 msgr2=0x7ff7d4198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:38.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 0x7ff7d4198030 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8006010 tx=0x7ff7c8004730 comp rx=0 tx=0).stop 2023-07-14T18:10:38.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 shutdown_connections 2023-07-14T18:10:38.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff7bc0a1150 0x7ff7bc0a3600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d4101850 0x7ff7d419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff7d40ffc10 0x7ff7d4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 --2- 172.21.15.115:0/235015282 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff7d4100ea0 0x7ff7d4198030 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:38.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 >> 172.21.15.115:0/235015282 conn(0x7ff7d40fb210 msgr2=0x7ff7d4106820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:38.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 shutdown_connections 2023-07-14T18:10:38.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.850+0000 7ff7d8959700 1 -- 172.21.15.115:0/235015282 wait complete. 2023-07-14T18:10:38.900 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-07-14T18:10:39.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.987+0000 7feedf58a700 1 -- 172.21.15.115:0/3490469783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 msgr2=0x7feed8101260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.987+0000 7feedf58a700 1 --2- 172.21.15.115:0/3490469783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 0x7feed8101260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feec8009990 tx=0x7feec8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.987+0000 7feedf58a700 1 -- 172.21.15.115:0/3490469783 shutdown_connections 2023-07-14T18:10:39.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.987+0000 7feedf58a700 1 --2- 172.21.15.115:0/3490469783 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feed81017a0 0x7feed8108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.987+0000 7feedf58a700 1 --2- 172.21.15.115:0/3490469783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feed80ffc20 0x7feed8100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.987+0000 7feedf58a700 1 --2- 172.21.15.115:0/3490469783 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 0x7feed8101260 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.987+0000 7feedf58a700 1 -- 172.21.15.115:0/3490469783 >> 172.21.15.115:0/3490469783 conn(0x7feed80fb260 msgr2=0x7feed80fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:39.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.987+0000 7feedf58a700 1 -- 172.21.15.115:0/3490469783 shutdown_connections 2023-07-14T18:10:39.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.987+0000 7feedf58a700 1 -- 172.21.15.115:0/3490469783 wait complete. 2023-07-14T18:10:39.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedf58a700 1 Processor -- start 2023-07-14T18:10:39.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedf58a700 1 -- start start 2023-07-14T18:10:39.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedf58a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feed80ffc20 0x7feed81958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedf58a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 0x7feed8195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedf58a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feed81017a0 0x7feed819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedf58a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed819d430 con 0x7feed8100e10 2023-07-14T18:10:39.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedf58a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed819d570 con 0x7feed81017a0 2023-07-14T18:10:39.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedf58a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed819d700 con 0x7feed80ffc20 2023-07-14T18:10:39.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedcb25700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 0x7feed8195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedcb25700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 0x7feed8195df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53036/0 (socket says 172.21.15.115:53036) 2023-07-14T18:10:39.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedcb25700 1 -- 172.21.15.115:0/3188008345 learned_addr learned my addr 172.21.15.115:0/3188008345 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:39.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feeddb27700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feed81017a0 0x7feed819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.988+0000 7feedd326700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feed80ffc20 0x7feed81958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feedcb25700 1 -- 172.21.15.115:0/3188008345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feed80ffc20 msgr2=0x7feed81958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feedcb25700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feed80ffc20 0x7feed81958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feedcb25700 1 -- 172.21.15.115:0/3188008345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feed81017a0 msgr2=0x7feed819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feedcb25700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feed81017a0 0x7feed819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feedcb25700 1 -- 172.21.15.115:0/3188008345 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec80095f0 con 0x7feed8100e10 2023-07-14T18:10:39.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feedcb25700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 0x7feed8195df0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7feec8003a50 tx=0x7feec8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:39.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feece7fc700 1 -- 172.21.15.115:0/3188008345 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec801d070 con 0x7feed8100e10 2023-07-14T18:10:39.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feece7fc700 1 -- 172.21.15.115:0/3188008345 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7feec8005a00 con 0x7feed8100e10 2023-07-14T18:10:39.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed819d980 con 0x7feed8100e10 2023-07-14T18:10:39.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feece7fc700 1 -- 172.21.15.115:0/3188008345 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec8005020 con 0x7feed8100e10 2023-07-14T18:10:39.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.989+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed819de20 con 0x7feed8100e10 2023-07-14T18:10:39.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.990+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed8106c00 con 0x7feed8100e10 2023-07-14T18:10:39.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.991+0000 7feece7fc700 1 -- 172.21.15.115:0/3188008345 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7feec8004800 con 0x7feed8100e10 2023-07-14T18:10:39.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.992+0000 7feece7fc700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feec40a11d0 0x7feec40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.992+0000 7feedd326700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feec40a11d0 0x7feec40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.992+0000 7feece7fc700 1 -- 172.21.15.115:0/3188008345 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7feec80cdb80 con 0x7feed8100e10 2023-07-14T18:10:39.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.992+0000 7feedd326700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feec40a11d0 0x7feec40a3680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feed8100c70 tx=0x7feed400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:39.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:38.996+0000 7feece7fc700 1 -- 172.21.15.115:0/3188008345 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7feec805b780 con 0x7feed8100e10 2023-07-14T18:10:39.105 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:10:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:10:38.807+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:10:39.106 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:10:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:10:38.807+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:10:39.106 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:10:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:10:38.807+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:10:39.106 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:10:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:10:38.807+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:10:39.106 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:10:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:10:38.807+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:10:39.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.218+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7feed8060a80 con 0x7feec40a11d0 2023-07-14T18:10:39.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.222+0000 7feece7fc700 1 -- 172.21.15.115:0/3188008345 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7feed8060a80 con 0x7feec40a11d0 2023-07-14T18:10:39.261 INFO:teuthology.orchestra.run.smithi115.stdout:/volumes/_nogroup/sv_0/0ce1c3df-0ddd-4436-9361-8f440b239633 2023-07-14T18:10:39.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feec40a11d0 msgr2=0x7feec40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feec40a11d0 0x7feec40a3680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7feed8100c70 tx=0x7feed400b410 comp rx=0 tx=0).stop 2023-07-14T18:10:39.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 msgr2=0x7feed8195df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 0x7feed8195df0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7feec8003a50 tx=0x7feec8009580 comp rx=0 tx=0).stop 2023-07-14T18:10:39.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 shutdown_connections 2023-07-14T18:10:39.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feec40a11d0 0x7feec40a3680 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feed81017a0 0x7feed819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feed80ffc20 0x7feed81958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 --2- 172.21.15.115:0/3188008345 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feed8100e10 0x7feed8195df0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 >> 172.21.15.115:0/3188008345 conn(0x7feed80fb260 msgr2=0x7feed80fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 shutdown_connections 2023-07-14T18:10:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.224+0000 7feedf58a700 1 -- 172.21.15.115:0/3188008345 wait complete. 2023-07-14T18:10:39.275 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-07-14T18:10:39.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.362+0000 7f55bf300700 1 -- 172.21.15.115:0/3876506927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80feb10 msgr2=0x7f55b80fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.362+0000 7f55bf300700 1 --2- 172.21.15.115:0/3876506927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80feb10 0x7f55b80fef20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009990 tx=0x7f55b4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.362+0000 7f55bf300700 1 -- 172.21.15.115:0/3876506927 shutdown_connections 2023-07-14T18:10:39.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.362+0000 7f55bf300700 1 --2- 172.21.15.115:0/3876506927 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55b8100690 0x7f55b8106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.362+0000 7f55bf300700 1 --2- 172.21.15.115:0/3876506927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80feb10 0x7f55b80fef20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.362+0000 7f55bf300700 1 --2- 172.21.15.115:0/3876506927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b80ffd00 0x7f55b8100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.362+0000 7f55bf300700 1 -- 172.21.15.115:0/3876506927 >> 172.21.15.115:0/3876506927 conn(0x7f55b80fa130 msgr2=0x7f55b80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:39.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 -- 172.21.15.115:0/3876506927 shutdown_connections 2023-07-14T18:10:39.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 -- 172.21.15.115:0/3876506927 wait complete. 2023-07-14T18:10:39.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 Processor -- start 2023-07-14T18:10:39.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 -- start start 2023-07-14T18:10:39.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55b80feb10 0x7f55b8194870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80ffd00 0x7f55b8194db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b8100690 0x7f55b819be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b819c3f0 con 0x7f55b80ffd00 2023-07-14T18:10:39.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b819c530 con 0x7f55b8100690 2023-07-14T18:10:39.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bf300700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b819c6c0 con 0x7f55b80feb10 2023-07-14T18:10:39.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bc89b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80ffd00 0x7f55b8194db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bc89b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80ffd00 0x7f55b8194db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53050/0 (socket says 172.21.15.115:53050) 2023-07-14T18:10:39.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.363+0000 7f55bc89b700 1 -- 172.21.15.115:0/518387670 learned_addr learned my addr 172.21.15.115:0/518387670 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:39.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bd09c700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55b80feb10 0x7f55b8194870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bd89d700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b8100690 0x7f55b819be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bc89b700 1 -- 172.21.15.115:0/518387670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55b80feb10 msgr2=0x7f55b8194870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bc89b700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55b80feb10 0x7f55b8194870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bc89b700 1 -- 172.21.15.115:0/518387670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b8100690 msgr2=0x7f55b819be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bc89b700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b8100690 0x7f55b819be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bc89b700 1 -- 172.21.15.115:0/518387670 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b40095f0 con 0x7f55b80ffd00 2023-07-14T18:10:39.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bd89d700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b8100690 0x7f55b819be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:39.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bc89b700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80ffd00 0x7f55b8194db0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f55a800c920 tx=0x7f55a800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:39.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55ae7fc700 1 -- 172.21.15.115:0/518387670 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8007b00 con 0x7f55b80ffd00 2023-07-14T18:10:39.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b819c9a0 con 0x7f55b80ffd00 2023-07-14T18:10:39.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55ae7fc700 1 -- 172.21.15.115:0/518387670 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f55a8004cc0 con 0x7f55b80ffd00 2023-07-14T18:10:39.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55ae7fc700 1 -- 172.21.15.115:0/518387670 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8005590 con 0x7f55b80ffd00 2023-07-14T18:10:39.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.364+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b819cf30 con 0x7f55b80ffd00 2023-07-14T18:10:39.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.365+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b810a260 con 0x7f55b80ffd00 2023-07-14T18:10:39.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.367+0000 7f55ae7fc700 1 -- 172.21.15.115:0/518387670 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f55a8004750 con 0x7f55b80ffd00 2023-07-14T18:10:39.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.367+0000 7f55ae7fc700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f55a40a1150 0x7f55a40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.367+0000 7f55bd09c700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f55a40a1150 0x7f55a40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.367+0000 7f55ae7fc700 1 -- 172.21.15.115:0/518387670 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f55a80cca70 con 0x7f55b80ffd00 2023-07-14T18:10:39.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.368+0000 7f55bd09c700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f55a40a1150 0x7f55a40a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f55b400b5c0 tx=0x7f55b4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:39.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.371+0000 7f55ae7fc700 1 -- 172.21.15.115:0/518387670 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55a8095fd0 con 0x7f55b80ffd00 2023-07-14T18:10:39.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:39 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] up:active 2023-07-14T18:10:39.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:39 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/903433874' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T18:10:39.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:39 smithi115 ceph-mon[127451]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2023-07-14T18:10:39.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:39 smithi115 ceph-mon[127451]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T18:10:39.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:39 smithi115 ceph-mon[127451]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:39.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:39 smithi115 ceph-mon[127451]: pgmap v262: 129 pgs: 129 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.5 KiB/s wr, 3 op/s 2023-07-14T18:10:39.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:39 smithi115 ceph-mon[127451]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:39.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.593+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f55b8060a80 con 0x7f55a40a1150 2023-07-14T18:10:39.656 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.617+0000 7f55ae7fc700 1 -- 172.21.15.115:0/518387670 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f55b8060a80 con 0x7f55a40a1150 2023-07-14T18:10:39.658 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f55a40a1150 msgr2=0x7f55a40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f55a40a1150 0x7f55a40a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f55b400b5c0 tx=0x7f55b4005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80ffd00 msgr2=0x7f55b8194db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80ffd00 0x7f55b8194db0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f55a800c920 tx=0x7f55a800cce0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 shutdown_connections 2023-07-14T18:10:39.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f55a40a1150 0x7f55a40a3600 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b8100690 0x7f55b819be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55b80feb10 0x7f55b8194870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 --2- 172.21.15.115:0/518387670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f55b80ffd00 0x7f55b8194db0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 >> 172.21.15.115:0/518387670 conn(0x7f55b80fa130 msgr2=0x7f55b80fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:39.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 shutdown_connections 2023-07-14T18:10:39.660 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.620+0000 7f55bf300700 1 -- 172.21.15.115:0/518387670 wait complete. 2023-07-14T18:10:39.671 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-07-14T18:10:39.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.756+0000 7f74c76d3700 1 -- 172.21.15.115:0/676102971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00feb10 msgr2=0x7f74c00fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.756+0000 7f74c76d3700 1 --2- 172.21.15.115:0/676102971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00feb10 0x7f74c00fef20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009990 tx=0x7f74bc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.756+0000 7f74c76d3700 1 -- 172.21.15.115:0/676102971 shutdown_connections 2023-07-14T18:10:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.756+0000 7f74c76d3700 1 --2- 172.21.15.115:0/676102971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74c0100690 0x7f74c0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.756+0000 7f74c76d3700 1 --2- 172.21.15.115:0/676102971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00feb10 0x7f74c00fef20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.756+0000 7f74c76d3700 1 --2- 172.21.15.115:0/676102971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c00ffd00 0x7f74c0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.756+0000 7f74c76d3700 1 -- 172.21.15.115:0/676102971 >> 172.21.15.115:0/676102971 conn(0x7f74c00fa130 msgr2=0x7f74c00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:39.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.756+0000 7f74c76d3700 1 -- 172.21.15.115:0/676102971 shutdown_connections 2023-07-14T18:10:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.756+0000 7f74c76d3700 1 -- 172.21.15.115:0/676102971 wait complete. 2023-07-14T18:10:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c76d3700 1 Processor -- start 2023-07-14T18:10:39.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c76d3700 1 -- start start 2023-07-14T18:10:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c76d3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74c00feb10 0x7f74c0194870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c76d3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00ffd00 0x7f74c0194db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c76d3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c0100690 0x7f74c019be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c76d3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c019c3f0 con 0x7f74c00ffd00 2023-07-14T18:10:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c76d3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c019c530 con 0x7f74c00feb10 2023-07-14T18:10:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c76d3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c019c6c0 con 0x7f74c0100690 2023-07-14T18:10:39.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c4c6e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00ffd00 0x7f74c0194db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c4c6e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00ffd00 0x7f74c0194db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53066/0 (socket says 172.21.15.115:53066) 2023-07-14T18:10:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c4c6e700 1 -- 172.21.15.115:0/3615119045 learned_addr learned my addr 172.21.15.115:0/3615119045 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.757+0000 7f74c546f700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74c00feb10 0x7f74c0194870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74c5c70700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c0100690 0x7f74c019be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74c4c6e700 1 -- 172.21.15.115:0/3615119045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c0100690 msgr2=0x7f74c019be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74c4c6e700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c0100690 0x7f74c019be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74c4c6e700 1 -- 172.21.15.115:0/3615119045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74c00feb10 msgr2=0x7f74c0194870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74c4c6e700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74c00feb10 0x7f74c0194870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74c4c6e700 1 -- 172.21.15.115:0/3615119045 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc0095f0 con 0x7f74c00ffd00 2023-07-14T18:10:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74c4c6e700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00ffd00 0x7f74c0194db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f74b000d950 tx=0x7f74b000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74b67fc700 1 -- 172.21.15.115:0/3615119045 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0009940 con 0x7f74c00ffd00 2023-07-14T18:10:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74b67fc700 1 -- 172.21.15.115:0/3615119045 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f74b0010460 con 0x7f74c00ffd00 2023-07-14T18:10:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74b67fc700 1 -- 172.21.15.115:0/3615119045 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b000f620 con 0x7f74c00ffd00 2023-07-14T18:10:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c019c9a0 con 0x7f74c00ffd00 2023-07-14T18:10:39.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.758+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c019cee0 con 0x7f74c00ffd00 2023-07-14T18:10:39.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.759+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c0105ac0 con 0x7f74c00ffd00 2023-07-14T18:10:39.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.760+0000 7f74b67fc700 1 -- 172.21.15.115:0/3615119045 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f74b00105d0 con 0x7f74c00ffd00 2023-07-14T18:10:39.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.761+0000 7f74b67fc700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f74ac0a1150 0x7f74ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:39.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.761+0000 7f74c546f700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f74ac0a1150 0x7f74ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:39.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.761+0000 7f74b67fc700 1 -- 172.21.15.115:0/3615119045 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f74b00cc8f0 con 0x7f74c00ffd00 2023-07-14T18:10:39.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.761+0000 7f74c546f700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f74ac0a1150 0x7f74ac0a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00c010 tx=0x7f74bc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:39.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.765+0000 7f74b67fc700 1 -- 172.21.15.115:0/3615119045 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f74b0095e50 con 0x7f74c00ffd00 2023-07-14T18:10:39.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:39 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] up:active 2023-07-14T18:10:39.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:39 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/903433874' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T18:10:39.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:39 smithi150 ceph-mon[128853]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2023-07-14T18:10:39.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:39 smithi150 ceph-mon[128853]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T18:10:39.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:39 smithi150 ceph-mon[128853]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:39.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:39 smithi150 ceph-mon[128853]: pgmap v262: 129 pgs: 129 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.5 KiB/s wr, 3 op/s 2023-07-14T18:10:39.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:39 smithi150 ceph-mon[128853]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:39.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:39 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] up:active 2023-07-14T18:10:39.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:39 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/903433874' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T18:10:39.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:39 smithi187 ceph-mon[129317]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2023-07-14T18:10:39.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:39 smithi187 ceph-mon[129317]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T18:10:39.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:39 smithi187 ceph-mon[129317]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:39.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:39 smithi187 ceph-mon[129317]: pgmap v262: 129 pgs: 129 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.5 KiB/s wr, 3 op/s 2023-07-14T18:10:39.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:39 smithi187 ceph-mon[129317]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:40.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.987+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f74c0060a80 con 0x7f74ac0a1150 2023-07-14T18:10:40.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.991+0000 7f74b67fc700 1 -- 172.21.15.115:0/3615119045 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f74c0060a80 con 0x7f74ac0a1150 2023-07-14T18:10:40.029 INFO:teuthology.orchestra.run.smithi115.stdout:/volumes/_nogroup/sv_1/15dcfb84-25af-4e0d-87ba-656292cf03b8 2023-07-14T18:10:40.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f74ac0a1150 msgr2=0x7f74ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f74ac0a1150 0x7f74ac0a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00c010 tx=0x7f74bc005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00ffd00 msgr2=0x7f74c0194db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00ffd00 0x7f74c0194db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f74b000d950 tx=0x7f74b000dc60 comp rx=0 tx=0).stop 2023-07-14T18:10:40.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 shutdown_connections 2023-07-14T18:10:40.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f74ac0a1150 0x7f74ac0a3600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f74c0100690 0x7f74c019be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74c00feb10 0x7f74c0194870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 --2- 172.21.15.115:0/3615119045 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f74c00ffd00 0x7f74c0194db0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 >> 172.21.15.115:0/3615119045 conn(0x7f74c00fa130 msgr2=0x7f74c00fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:40.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 shutdown_connections 2023-07-14T18:10:40.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:39.993+0000 7f74c76d3700 1 -- 172.21.15.115:0/3615119045 wait complete. 2023-07-14T18:10:40.043 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:10:40.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.126+0000 7f87767a4700 1 -- 172.21.15.115:0/2254329216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770100700 msgr2=0x7f8770102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.126+0000 7f87767a4700 1 --2- 172.21.15.115:0/2254329216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770100700 0x7f8770102ae0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8760009990 tx=0x7f8760009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.126+0000 7f87767a4700 1 -- 172.21.15.115:0/2254329216 shutdown_connections 2023-07-14T18:10:40.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.126+0000 7f87767a4700 1 --2- 172.21.15.115:0/2254329216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8770105940 0x7f8770107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.126+0000 7f87767a4700 1 --2- 172.21.15.115:0/2254329216 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770100700 0x7f8770102ae0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.126+0000 7f87767a4700 1 --2- 172.21.15.115:0/2254329216 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8770103020 0x7f8770105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.126+0000 7f87767a4700 1 -- 172.21.15.115:0/2254329216 >> 172.21.15.115:0/2254329216 conn(0x7f87700fa100 msgr2=0x7f87700fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:40.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.126+0000 7f87767a4700 1 -- 172.21.15.115:0/2254329216 shutdown_connections 2023-07-14T18:10:40.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.126+0000 7f87767a4700 1 -- 172.21.15.115:0/2254329216 wait complete. 2023-07-14T18:10:40.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f87767a4700 1 Processor -- start 2023-07-14T18:10:40.167 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f87767a4700 1 -- start start 2023-07-14T18:10:40.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f87767a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8770100700 0x7f8770195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f87767a4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770103020 0x7f8770195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f87767a4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8770105940 0x7f877019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f87767a4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877019d3c0 con 0x7f8770103020 2023-07-14T18:10:40.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f87767a4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877019d500 con 0x7f8770105940 2023-07-14T18:10:40.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f87767a4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877019d690 con 0x7f8770100700 2023-07-14T18:10:40.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8767fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770103020 0x7f8770195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8767fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770103020 0x7f8770195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53080/0 (socket says 172.21.15.115:53080) 2023-07-14T18:10:40.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8767fff700 1 -- 172.21.15.115:0/644501502 learned_addr learned my addr 172.21.15.115:0/644501502 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:40.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8774d41700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8770105940 0x7f877019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f876ffff700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8770100700 0x7f8770195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8767fff700 1 -- 172.21.15.115:0/644501502 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8770100700 msgr2=0x7f8770195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8767fff700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8770100700 0x7f8770195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8767fff700 1 -- 172.21.15.115:0/644501502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8770105940 msgr2=0x7f877019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8767fff700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8770105940 0x7f877019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8767fff700 1 -- 172.21.15.115:0/644501502 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87600095f0 con 0x7f8770103020 2023-07-14T18:10:40.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.127+0000 7f8774d41700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8770105940 0x7f877019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:40.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.128+0000 7f8767fff700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770103020 0x7f8770195d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f875800d950 tx=0x7f875800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:40.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.128+0000 7f876dffb700 1 -- 172.21.15.115:0/644501502 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758009940 con 0x7f8770103020 2023-07-14T18:10:40.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.128+0000 7f876dffb700 1 -- 172.21.15.115:0/644501502 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8758010460 con 0x7f8770103020 2023-07-14T18:10:40.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.128+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877019d970 con 0x7f8770103020 2023-07-14T18:10:40.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.128+0000 7f876dffb700 1 -- 172.21.15.115:0/644501502 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875800f620 con 0x7f8770103020 2023-07-14T18:10:40.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.128+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877019ded0 con 0x7f8770103020 2023-07-14T18:10:40.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.129+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877010b2b0 con 0x7f8770103020 2023-07-14T18:10:40.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.130+0000 7f876dffb700 1 -- 172.21.15.115:0/644501502 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f87580105d0 con 0x7f8770103020 2023-07-14T18:10:40.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.130+0000 7f876dffb700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f875c0a1220 0x7f875c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.130+0000 7f876ffff700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f875c0a1220 0x7f875c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.130+0000 7f876dffb700 1 -- 172.21.15.115:0/644501502 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f87580cc8b0 con 0x7f8770103020 2023-07-14T18:10:40.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.131+0000 7f876ffff700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f875c0a1220 0x7f875c0a36d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f876000b5c0 tx=0x7f8760009f00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:40.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.134+0000 7f876dffb700 1 -- 172.21.15.115:0/644501502 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8758095e10 con 0x7f8770103020 2023-07-14T18:10:40.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.368+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f877010b4a0 con 0x7f8770103020 2023-07-14T18:10:40.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.369+0000 7f876dffb700 1 -- 172.21.15.115:0/644501502 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10851 (secure 0 0 0) 0x7f8758016090 con 0x7f8770103020 2023-07-14T18:10:40.408 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:40.408 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:10:38.422616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24493,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24446":{"gid":24446,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:10:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f875c0a1220 msgr2=0x7f875c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f875c0a1220 0x7f875c0a36d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f876000b5c0 tx=0x7f8760009f00 comp rx=0 tx=0).stop 2023-07-14T18:10:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770103020 msgr2=0x7f8770195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770103020 0x7f8770195d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f875800d950 tx=0x7f875800dc60 comp rx=0 tx=0).stop 2023-07-14T18:10:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 shutdown_connections 2023-07-14T18:10:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f875c0a1220 0x7f875c0a36d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8770105940 0x7f877019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8770100700 0x7f8770195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 --2- 172.21.15.115:0/644501502 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8770103020 0x7f8770195d80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 >> 172.21.15.115:0/644501502 conn(0x7f87700fa100 msgr2=0x7f8770101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 shutdown_connections 2023-07-14T18:10:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.371+0000 7f87767a4700 1 -- 172.21.15.115:0/644501502 wait complete. 2023-07-14T18:10:40.412 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 24 2023-07-14T18:10:40.422 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:10:40.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.504+0000 7f317cca0700 1 -- 172.21.15.115:0/4013284372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 msgr2=0x7f3178107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.504+0000 7f317cca0700 1 --2- 172.21.15.115:0/4013284372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 0x7f3178107d50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f317000b0a0 tx=0x7f317000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.504+0000 7f317cca0700 1 -- 172.21.15.115:0/4013284372 shutdown_connections 2023-07-14T18:10:40.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.504+0000 7f317cca0700 1 --2- 172.21.15.115:0/4013284372 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 0x7f3178107d50 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.504+0000 7f317cca0700 1 --2- 172.21.15.115:0/4013284372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f31781006b0 0x7f3178102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.504+0000 7f317cca0700 1 --2- 172.21.15.115:0/4013284372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3178102fd0 0x7f31781053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.504+0000 7f317cca0700 1 -- 172.21.15.115:0/4013284372 >> 172.21.15.115:0/4013284372 conn(0x7f31780fa150 msgr2=0x7f31780fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:40.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.504+0000 7f317cca0700 1 -- 172.21.15.115:0/4013284372 shutdown_connections 2023-07-14T18:10:40.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.504+0000 7f317cca0700 1 -- 172.21.15.115:0/4013284372 wait complete. 2023-07-14T18:10:40.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f317cca0700 1 Processor -- start 2023-07-14T18:10:40.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f317cca0700 1 -- start start 2023-07-14T18:10:40.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f317cca0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31781006b0 0x7f3178100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f317cca0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3178102fd0 0x7f31780fe660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f317cca0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 0x7f31780feba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f317cca0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31780ff340 con 0x7f31781058f0 2023-07-14T18:10:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f317cca0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31780ff4b0 con 0x7f31781006b0 2023-07-14T18:10:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f317cca0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317819f970 con 0x7f3178102fd0 2023-07-14T18:10:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f3176d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 0x7f31780feba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f3176d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 0x7f31780feba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53096/0 (socket says 172.21.15.115:53096) 2023-07-14T18:10:40.547 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f3176d9d700 1 -- 172.21.15.115:0/3551278773 learned_addr learned my addr 172.21.15.115:0/3551278773 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.505+0000 7f3175d9b700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3178102fd0 0x7f31780fe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f317659c700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31781006b0 0x7f3178100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f3176d9d700 1 -- 172.21.15.115:0/3551278773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3178102fd0 msgr2=0x7f31780fe660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f3176d9d700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3178102fd0 0x7f31780fe660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f3176d9d700 1 -- 172.21.15.115:0/3551278773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31781006b0 msgr2=0x7f3178100020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f3176d9d700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31781006b0 0x7f3178100020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f3176d9d700 1 -- 172.21.15.115:0/3551278773 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3170009d00 con 0x7f31781058f0 2023-07-14T18:10:40.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f3176d9d700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 0x7f31780feba0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f317000b070 tx=0x7f3170004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:40.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f316f7fe700 1 -- 172.21.15.115:0/3551278773 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317000e040 con 0x7f31781058f0 2023-07-14T18:10:40.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317819fbf0 con 0x7f31781058f0 2023-07-14T18:10:40.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f316f7fe700 1 -- 172.21.15.115:0/3551278773 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3170003e90 con 0x7f31781058f0 2023-07-14T18:10:40.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f316f7fe700 1 -- 172.21.15.115:0/3551278773 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170004020 con 0x7f31781058f0 2023-07-14T18:10:40.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.506+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31781a0070 con 0x7f31781058f0 2023-07-14T18:10:40.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.508+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31780fbcc0 con 0x7f31781058f0 2023-07-14T18:10:40.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.508+0000 7f316f7fe700 1 -- 172.21.15.115:0/3551278773 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f317001b660 con 0x7f31781058f0 2023-07-14T18:10:40.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.509+0000 7f316f7fe700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f31640a11a0 0x7f31640a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.509+0000 7f317659c700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f31640a11a0 0x7f31640a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.509+0000 7f316f7fe700 1 -- 172.21.15.115:0/3551278773 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f31700ce470 con 0x7f31781058f0 2023-07-14T18:10:40.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.509+0000 7f317659c700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f31640a11a0 0x7f31640a3650 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3168005950 tx=0x7f31680058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:40.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.513+0000 7f316f7fe700 1 -- 172.21.15.115:0/3551278773 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3170097a60 con 0x7f31781058f0 2023-07-14T18:10:40.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.705+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3178061250 con 0x7f31781058f0 2023-07-14T18:10:40.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.707+0000 7f316f7fe700 1 -- 172.21.15.115:0/3551278773 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7f317009b080 con 0x7f31781058f0 2023-07-14T18:10:40.746 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:40.746 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f31640a11a0 msgr2=0x7f31640a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f31640a11a0 0x7f31640a3650 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3168005950 tx=0x7f31680058e0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 msgr2=0x7f31780feba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 0x7f31780feba0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f317000b070 tx=0x7f3170004c80 comp rx=0 tx=0).stop 2023-07-14T18:10:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 shutdown_connections 2023-07-14T18:10:40.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f31640a11a0 0x7f31640a3650 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f31781058f0 0x7f31780feba0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31781006b0 0x7f3178100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 --2- 172.21.15.115:0/3551278773 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3178102fd0 0x7f31780fe660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 >> 172.21.15.115:0/3551278773 conn(0x7f31780fa150 msgr2=0x7f3178101310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:40.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 shutdown_connections 2023-07-14T18:10:40.750 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.709+0000 7f317cca0700 1 -- 172.21.15.115:0/3551278773 wait complete. 2023-07-14T18:10:40.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:40 smithi115 ceph-mon[127451]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:40.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:40 smithi115 ceph-mon[127451]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:40.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:40 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/644501502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:40.760 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-14T18:10:40.761 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T18:10:40.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:40 smithi150 ceph-mon[128853]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:40.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:40 smithi150 ceph-mon[128853]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:40.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:40 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/644501502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:40.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.843+0000 7f27c15e1700 1 -- 172.21.15.115:0/630623821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc1006a0 msgr2=0x7f27bc106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.843+0000 7f27c15e1700 1 --2- 172.21.15.115:0/630623821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc1006a0 0x7f27bc106f60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f27b400b0a0 tx=0x7f27b400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.843+0000 7f27c15e1700 1 -- 172.21.15.115:0/630623821 shutdown_connections 2023-07-14T18:10:40.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.843+0000 7f27c15e1700 1 --2- 172.21.15.115:0/630623821 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc1006a0 0x7f27bc106f60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.843+0000 7f27c15e1700 1 --2- 172.21.15.115:0/630623821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc0feb20 0x7f27bc0fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.843+0000 7f27c15e1700 1 --2- 172.21.15.115:0/630623821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0ffd10 0x7f27bc100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.882 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.843+0000 7f27c15e1700 1 -- 172.21.15.115:0/630623821 >> 172.21.15.115:0/630623821 conn(0x7f27bc0fa100 msgr2=0x7f27bc0fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:40.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.843+0000 7f27c15e1700 1 -- 172.21.15.115:0/630623821 shutdown_connections 2023-07-14T18:10:40.883 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.843+0000 7f27c15e1700 1 -- 172.21.15.115:0/630623821 wait complete. 2023-07-14T18:10:40.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27c15e1700 1 Processor -- start 2023-07-14T18:10:40.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27c15e1700 1 -- start start 2023-07-14T18:10:40.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27c15e1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc0feb20 0x7f27bc1948c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27c15e1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0ffd10 0x7f27bc194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27c15e1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc1006a0 0x7f27bc19be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27c15e1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc19c440 con 0x7f27bc0feb20 2023-07-14T18:10:40.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27c15e1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc19c580 con 0x7f27bc0ffd10 2023-07-14T18:10:40.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27c15e1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc19c710 con 0x7f27bc1006a0 2023-07-14T18:10:40.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc0feb20 0x7f27bc1948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27baffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc0feb20 0x7f27bc1948c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53106/0 (socket says 172.21.15.115:53106) 2023-07-14T18:10:40.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27baffd700 1 -- 172.21.15.115:0/3495456507 learned_addr learned my addr 172.21.15.115:0/3495456507 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:40.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27ba7fc700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0ffd10 0x7f27bc194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27baffd700 1 -- 172.21.15.115:0/3495456507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc1006a0 msgr2=0x7f27bc19be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27baffd700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc1006a0 0x7f27bc19be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27baffd700 1 -- 172.21.15.115:0/3495456507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0ffd10 msgr2=0x7f27bc194e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:40.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27baffd700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0ffd10 0x7f27bc194e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:40.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.844+0000 7f27baffd700 1 -- 172.21.15.115:0/3495456507 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b4009d00 con 0x7f27bc0feb20 2023-07-14T18:10:40.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.845+0000 7f27baffd700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc0feb20 0x7f27bc1948c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f27a40098f0 tx=0x7f27a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:40.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.845+0000 7f27b3fff700 1 -- 172.21.15.115:0/3495456507 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4012070 con 0x7f27bc0feb20 2023-07-14T18:10:40.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.845+0000 7f27b3fff700 1 -- 172.21.15.115:0/3495456507 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f27a4004020 con 0x7f27bc0feb20 2023-07-14T18:10:40.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.845+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc068830 con 0x7f27bc0feb20 2023-07-14T18:10:40.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.845+0000 7f27b3fff700 1 -- 172.21.15.115:0/3495456507 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a4003cc0 con 0x7f27bc0feb20 2023-07-14T18:10:40.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.845+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc068d40 con 0x7f27bc0feb20 2023-07-14T18:10:40.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.847+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc066ba0 con 0x7f27bc0feb20 2023-07-14T18:10:40.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.847+0000 7f27b3fff700 1 -- 172.21.15.115:0/3495456507 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f27a4016030 con 0x7f27bc0feb20 2023-07-14T18:10:40.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.848+0000 7f27b3fff700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f27a80a12d0 0x7f27a80a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:40.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.848+0000 7f27b3fff700 1 -- 172.21.15.115:0/3495456507 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f27a40cdb80 con 0x7f27bc0feb20 2023-07-14T18:10:40.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.848+0000 7f27ba7fc700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f27a80a12d0 0x7f27a80a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:40.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.848+0000 7f27ba7fc700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f27a80a12d0 0x7f27a80a3780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f27ac005950 tx=0x7f27ac00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:40.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:40.852+0000 7f27b3fff700 1 -- 172.21.15.115:0/3495456507 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27a40970e0 con 0x7f27bc0feb20 2023-07-14T18:10:40.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:40 smithi187 ceph-mon[129317]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:40.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:40 smithi187 ceph-mon[129317]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:10:40.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:40 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/644501502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:10:41.078 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.039+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27bc068fc0 con 0x7f27a80a12d0 2023-07-14T18:10:41.090 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.052+0000 7f27b3fff700 1 -- 172.21.15.115:0/3495456507 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418901 (secure 0 0 0) 0x7f27bc068fc0 con 0x7f27a80a12d0 2023-07-14T18:10:41.095 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:41.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.055+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f27a80a12d0 msgr2=0x7f27a80a3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:41.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.055+0000 7f27c15e1700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f27a80a12d0 0x7f27a80a3780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f27ac005950 tx=0x7f27ac00a300 comp rx=0 tx=0).stop 2023-07-14T18:10:41.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.055+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc0feb20 msgr2=0x7f27bc1948c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:41.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.055+0000 7f27c15e1700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc0feb20 0x7f27bc1948c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f27a40098f0 tx=0x7f27a4009c00 comp rx=0 tx=0).stop 2023-07-14T18:10:41.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.055+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 shutdown_connections 2023-07-14T18:10:41.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.055+0000 7f27c15e1700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f27a80a12d0 0x7f27a80a3780 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.055+0000 7f27c15e1700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc1006a0 0x7f27bc19be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.056+0000 7f27c15e1700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27bc0feb20 0x7f27bc1948c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.056+0000 7f27c15e1700 1 --2- 172.21.15.115:0/3495456507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc0ffd10 0x7f27bc194e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.056+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 >> 172.21.15.115:0/3495456507 conn(0x7f27bc0fa100 msgr2=0x7f27bc1031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:41.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.056+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 shutdown_connections 2023-07-14T18:10:41.099 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.056+0000 7f27c15e1700 1 -- 172.21.15.115:0/3495456507 wait complete. 2023-07-14T18:10:41.100 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-07-14T18:10:41.107 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2023-07-14T18:10:40.885199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596922,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":133,"num_read_kb":116,"num_write":277,"num_write_kb":2971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337420,"kb_used_data":13196,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604404,"statfs":{"total":1151940427776,"available":1151594909696,"internally_reserved":0,"allocated":13512704,"data_stored":7447753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4192,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":51,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003285"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089430+0000","last_change":"2023-07-14T18:10:18.110819+0000","last_active":"2023-07-14T18:10:24.089430+0000","last_peered":"2023-07-14T18:10:24.089430+0000","last_clean":"2023-07-14T18:10:24.089430+0000","last_became_active":"2023-07-14T18:10:18.110180+0000","last_became_peered":"2023-07-14T18:10:18.110180+0000","last_unstale":"2023-07-14T18:10:24.089430+0000","last_undegraded":"2023-07-14T18:10:24.089430+0000","last_fullsized":"2023-07-14T18:10:24.089430+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:08:50.579212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440800+0000","last_change":"2023-07-14T18:10:18.117539+0000","last_active":"2023-07-14T18:10:23.440800+0000","last_peered":"2023-07-14T18:10:23.440800+0000","last_clean":"2023-07-14T18:10:23.440800+0000","last_became_active":"2023-07-14T18:10:18.115956+0000","last_became_peered":"2023-07-14T18:10:18.115956+0000","last_unstale":"2023-07-14T18:10:23.440800+0000","last_undegraded":"2023-07-14T18:10:23.440800+0000","last_fullsized":"2023-07-14T18:10:23.440800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:22.723383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440427+0000","last_change":"2023-07-14T18:10:18.120582+0000","last_active":"2023-07-14T18:10:23.440427+0000","last_peered":"2023-07-14T18:10:23.440427+0000","last_clean":"2023-07-14T18:10:23.440427+0000","last_became_active":"2023-07-14T18:10:18.120296+0000","last_became_peered":"2023-07-14T18:10:18.120296+0000","last_unstale":"2023-07-14T18:10:23.440427+0000","last_undegraded":"2023-07-14T18:10:23.440427+0000","last_fullsized":"2023-07-14T18:10:23.440427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:54.381627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438612+0000","last_change":"2023-07-14T18:10:18.112250+0000","last_active":"2023-07-14T18:10:23.438612+0000","last_peered":"2023-07-14T18:10:23.438612+0000","last_clean":"2023-07-14T18:10:23.438612+0000","last_became_active":"2023-07-14T18:10:18.111514+0000","last_became_peered":"2023-07-14T18:10:18.111514+0000","last_unstale":"2023-07-14T18:10:23.438612+0000","last_undegraded":"2023-07-14T18:10:23.438612+0000","last_fullsized":"2023-07-14T18:10:23.438612+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:02:30.280559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438548+0000","last_change":"2023-07-14T18:10:18.111127+0000","last_active":"2023-07-14T18:10:23.438548+0000","last_peered":"2023-07-14T18:10:23.438548+0000","last_clean":"2023-07-14T18:10:23.438548+0000","last_became_active":"2023-07-14T18:10:18.110816+0000","last_became_peered":"2023-07-14T18:10:18.110816+0000","last_unstale":"2023-07-14T18:10:23.438548+0000","last_undegraded":"2023-07-14T18:10:23.438548+0000","last_fullsized":"2023-07-14T18:10:23.438548+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:05:59.027134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734705+0000","last_change":"2023-07-14T18:10:18.113436+0000","last_active":"2023-07-14T18:10:23.734705+0000","last_peered":"2023-07-14T18:10:23.734705+0000","last_clean":"2023-07-14T18:10:23.734705+0000","last_became_active":"2023-07-14T18:10:18.113172+0000","last_became_peered":"2023-07-14T18:10:18.113172+0000","last_unstale":"2023-07-14T18:10:23.734705+0000","last_undegraded":"2023-07-14T18:10:23.734705+0000","last_fullsized":"2023-07-14T18:10:23.734705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:33:57.446784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089034+0000","last_change":"2023-07-14T18:10:18.116720+0000","last_active":"2023-07-14T18:10:24.089034+0000","last_peered":"2023-07-14T18:10:24.089034+0000","last_clean":"2023-07-14T18:10:24.089034+0000","last_became_active":"2023-07-14T18:10:18.116361+0000","last_became_peered":"2023-07-14T18:10:18.116361+0000","last_unstale":"2023-07-14T18:10:24.089034+0000","last_undegraded":"2023-07-14T18:10:24.089034+0000","last_fullsized":"2023-07-14T18:10:24.089034+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:54.195458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088667+0000","last_change":"2023-07-14T18:10:18.111981+0000","last_active":"2023-07-14T18:10:24.088667+0000","last_peered":"2023-07-14T18:10:24.088667+0000","last_clean":"2023-07-14T18:10:24.088667+0000","last_became_active":"2023-07-14T18:10:18.111666+0000","last_became_peered":"2023-07-14T18:10:18.111666+0000","last_unstale":"2023-07-14T18:10:24.088667+0000","last_undegraded":"2023-07-14T18:10:24.088667+0000","last_fullsized":"2023-07-14T18:10:24.088667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:07:35.710429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089557+0000","last_change":"2023-07-14T18:10:18.119497+0000","last_active":"2023-07-14T18:10:24.089557+0000","last_peered":"2023-07-14T18:10:24.089557+0000","last_clean":"2023-07-14T18:10:24.089557+0000","last_became_active":"2023-07-14T18:10:18.119090+0000","last_became_peered":"2023-07-14T18:10:18.119090+0000","last_unstale":"2023-07-14T18:10:24.089557+0000","last_undegraded":"2023-07-14T18:10:24.089557+0000","last_fullsized":"2023-07-14T18:10:24.089557+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:15:13.934598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.735011+0000","last_change":"2023-07-14T18:10:18.107353+0000","last_active":"2023-07-14T18:10:23.735011+0000","last_peered":"2023-07-14T18:10:23.735011+0000","last_clean":"2023-07-14T18:10:23.735011+0000","last_became_active":"2023-07-14T18:10:18.106933+0000","last_became_peered":"2023-07-14T18:10:18.106933+0000","last_unstale":"2023-07-14T18:10:23.735011+0000","last_undegraded":"2023-07-14T18:10:23.735011+0000","last_fullsized":"2023-07-14T18:10:23.735011+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:20:25.508991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089317+0000","last_change":"2023-07-14T18:10:18.121064+0000","last_active":"2023-07-14T18:10:24.089317+0000","last_peered":"2023-07-14T18:10:24.089317+0000","last_clean":"2023-07-14T18:10:24.089317+0000","last_became_active":"2023-07-14T18:10:18.120758+0000","last_became_peered":"2023-07-14T18:10:18.120758+0000","last_unstale":"2023-07-14T18:10:24.089317+0000","last_undegraded":"2023-07-14T18:10:24.089317+0000","last_fullsized":"2023-07-14T18:10:24.089317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:42.972896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438568+0000","last_change":"2023-07-14T18:10:18.112436+0000","last_active":"2023-07-14T18:10:23.438568+0000","last_peered":"2023-07-14T18:10:23.438568+0000","last_clean":"2023-07-14T18:10:23.438568+0000","last_became_active":"2023-07-14T18:10:18.111942+0000","last_became_peered":"2023-07-14T18:10:18.111942+0000","last_unstale":"2023-07-14T18:10:23.438568+0000","last_undegraded":"2023-07-14T18:10:23.438568+0000","last_fullsized":"2023-07-14T18:10:23.438568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:57.856511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845884+0000","last_change":"2023-07-14T18:10:18.106134+0000","last_active":"2023-07-14T18:10:23.845884+0000","last_peered":"2023-07-14T18:10:23.845884+0000","last_clean":"2023-07-14T18:10:23.845884+0000","last_became_active":"2023-07-14T18:10:18.105481+0000","last_became_peered":"2023-07-14T18:10:18.105481+0000","last_unstale":"2023-07-14T18:10:23.845884+0000","last_undegraded":"2023-07-14T18:10:23.845884+0000","last_fullsized":"2023-07-14T18:10:23.845884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:53.958762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440197+0000","last_change":"2023-07-14T18:10:18.117399+0000","last_active":"2023-07-14T18:10:23.440197+0000","last_peered":"2023-07-14T18:10:23.440197+0000","last_clean":"2023-07-14T18:10:23.440197+0000","last_became_active":"2023-07-14T18:10:18.117094+0000","last_became_peered":"2023-07-14T18:10:18.117094+0000","last_unstale":"2023-07-14T18:10:23.440197+0000","last_undegraded":"2023-07-14T18:10:23.440197+0000","last_fullsized":"2023-07-14T18:10:23.440197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:10:09.445305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440157+0000","last_change":"2023-07-14T18:10:18.117150+0000","last_active":"2023-07-14T18:10:23.440157+0000","last_peered":"2023-07-14T18:10:23.440157+0000","last_clean":"2023-07-14T18:10:23.440157+0000","last_became_active":"2023-07-14T18:10:18.116805+0000","last_became_peered":"2023-07-14T18:10:18.116805+0000","last_unstale":"2023-07-14T18:10:23.440157+0000","last_undegraded":"2023-07-14T18:10:23.440157+0000","last_fullsized":"2023-07-14T18:10:23.440157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:48:43.803465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088549+0000","last_change":"2023-07-14T18:10:18.120868+0000","last_active":"2023-07-14T18:10:24.088549+0000","last_peered":"2023-07-14T18:10:24.088549+0000","last_clean":"2023-07-14T18:10:24.088549+0000","last_became_active":"2023-07-14T18:10:18.120083+0000","last_became_peered":"2023-07-14T18:10:18.120083+0000","last_unstale":"2023-07-14T18:10:24.088549+0000","last_undegraded":"2023-07-14T18:10:24.088549+0000","last_fullsized":"2023-07-14T18:10:24.088549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:50:07.162775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089298+0000","last_change":"2023-07-14T18:10:18.111011+0000","last_active":"2023-07-14T18:10:24.089298+0000","last_peered":"2023-07-14T18:10:24.089298+0000","last_clean":"2023-07-14T18:10:24.089298+0000","last_became_active":"2023-07-14T18:10:18.110517+0000","last_became_peered":"2023-07-14T18:10:18.110517+0000","last_unstale":"2023-07-14T18:10:24.089298+0000","last_undegraded":"2023-07-14T18:10:24.089298+0000","last_fullsized":"2023-07-14T18:10:24.089298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:00:45.278711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089798+0000","last_change":"2023-07-14T18:10:18.110739+0000","last_active":"2023-07-14T18:10:24.089798+0000","last_peered":"2023-07-14T18:10:24.089798+0000","last_clean":"2023-07-14T18:10:24.089798+0000","last_became_active":"2023-07-14T18:10:18.110442+0000","last_became_peered":"2023-07-14T18:10:18.110442+0000","last_unstale":"2023-07-14T18:10:24.089798+0000","last_undegraded":"2023-07-14T18:10:24.089798+0000","last_fullsized":"2023-07-14T18:10:24.089798+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:45.130540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550066+0000","last_change":"2023-07-14T18:10:18.109955+0000","last_active":"2023-07-14T18:10:23.550066+0000","last_peered":"2023-07-14T18:10:23.550066+0000","last_clean":"2023-07-14T18:10:23.550066+0000","last_became_active":"2023-07-14T18:10:18.109651+0000","last_became_peered":"2023-07-14T18:10:18.109651+0000","last_unstale":"2023-07-14T18:10:23.550066+0000","last_undegraded":"2023-07-14T18:10:23.550066+0000","last_fullsized":"2023-07-14T18:10:23.550066+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:22:46.479116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549977+0000","last_change":"2023-07-14T18:10:22.457996+0000","last_active":"2023-07-14T18:10:23.549977+0000","last_peered":"2023-07-14T18:10:23.549977+0000","last_clean":"2023-07-14T18:10:23.549977+0000","last_became_active":"2023-07-14T18:10:22.457573+0000","last_became_peered":"2023-07-14T18:10:22.457573+0000","last_unstale":"2023-07-14T18:10:23.549977+0000","last_undegraded":"2023-07-14T18:10:23.549977+0000","last_fullsized":"2023-07-14T18:10:23.549977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:17.148116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090026+0000","last_change":"2023-07-14T18:10:17.216780+0000","last_active":"2023-07-14T18:10:24.090026+0000","last_peered":"2023-07-14T18:10:24.090026+0000","last_clean":"2023-07-14T18:10:24.090026+0000","last_became_active":"2023-07-14T18:10:17.216423+0000","last_became_peered":"2023-07-14T18:10:17.216423+0000","last_unstale":"2023-07-14T18:10:24.090026+0000","last_undegraded":"2023-07-14T18:10:24.090026+0000","last_fullsized":"2023-07-14T18:10:24.090026+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:24.522131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089127+0000","last_change":"2023-07-14T18:10:18.108615+0000","last_active":"2023-07-14T18:10:24.089127+0000","last_peered":"2023-07-14T18:10:24.089127+0000","last_clean":"2023-07-14T18:10:24.089127+0000","last_became_active":"2023-07-14T18:10:18.108316+0000","last_became_peered":"2023-07-14T18:10:18.108316+0000","last_unstale":"2023-07-14T18:10:24.089127+0000","last_undegraded":"2023-07-14T18:10:24.089127+0000","last_fullsized":"2023-07-14T18:10:24.089127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:27:25.084342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438203+0000","last_change":"2023-07-14T18:10:22.453040+0000","last_active":"2023-07-14T18:10:23.438203+0000","last_peered":"2023-07-14T18:10:23.438203+0000","last_clean":"2023-07-14T18:10:23.438203+0000","last_became_active":"2023-07-14T18:10:22.452643+0000","last_became_peered":"2023-07-14T18:10:22.452643+0000","last_unstale":"2023-07-14T18:10:23.438203+0000","last_undegraded":"2023-07-14T18:10:23.438203+0000","last_fullsized":"2023-07-14T18:10:23.438203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:18.191433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089404+0000","last_change":"2023-07-14T18:10:17.214085+0000","last_active":"2023-07-14T18:10:24.089404+0000","last_peered":"2023-07-14T18:10:24.089404+0000","last_clean":"2023-07-14T18:10:24.089404+0000","last_became_active":"2023-07-14T18:10:17.213772+0000","last_became_peered":"2023-07-14T18:10:17.213772+0000","last_unstale":"2023-07-14T18:10:24.089404+0000","last_undegraded":"2023-07-14T18:10:24.089404+0000","last_fullsized":"2023-07-14T18:10:24.089404+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:16:35.586284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440705+0000","last_change":"2023-07-14T18:10:18.120532+0000","last_active":"2023-07-14T18:10:23.440705+0000","last_peered":"2023-07-14T18:10:23.440705+0000","last_clean":"2023-07-14T18:10:23.440705+0000","last_became_active":"2023-07-14T18:10:18.120199+0000","last_became_peered":"2023-07-14T18:10:18.120199+0000","last_unstale":"2023-07-14T18:10:23.440705+0000","last_undegraded":"2023-07-14T18:10:23.440705+0000","last_fullsized":"2023-07-14T18:10:23.440705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:05:38.292570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438439+0000","last_change":"2023-07-14T18:10:22.458820+0000","last_active":"2023-07-14T18:10:23.438439+0000","last_peered":"2023-07-14T18:10:23.438439+0000","last_clean":"2023-07-14T18:10:23.438439+0000","last_became_active":"2023-07-14T18:10:22.458480+0000","last_became_peered":"2023-07-14T18:10:22.458480+0000","last_unstale":"2023-07-14T18:10:23.438439+0000","last_undegraded":"2023-07-14T18:10:23.438439+0000","last_fullsized":"2023-07-14T18:10:23.438439+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:45.901536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428015+0000","last_change":"2023-07-14T18:10:17.212329+0000","last_active":"2023-07-14T18:10:37.428015+0000","last_peered":"2023-07-14T18:10:37.428015+0000","last_clean":"2023-07-14T18:10:37.428015+0000","last_became_active":"2023-07-14T18:10:17.212013+0000","last_became_peered":"2023-07-14T18:10:17.212013+0000","last_unstale":"2023-07-14T18:10:37.428015+0000","last_undegraded":"2023-07-14T18:10:37.428015+0000","last_fullsized":"2023-07-14T18:10:37.428015+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:19:15.950647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089229+0000","last_change":"2023-07-14T18:10:18.114621+0000","last_active":"2023-07-14T18:10:24.089229+0000","last_peered":"2023-07-14T18:10:24.089229+0000","last_clean":"2023-07-14T18:10:24.089229+0000","last_became_active":"2023-07-14T18:10:18.114330+0000","last_became_peered":"2023-07-14T18:10:18.114330+0000","last_unstale":"2023-07-14T18:10:24.089229+0000","last_undegraded":"2023-07-14T18:10:24.089229+0000","last_fullsized":"2023-07-14T18:10:24.089229+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:47:21.060539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438244+0000","last_change":"2023-07-14T18:10:22.453179+0000","last_active":"2023-07-14T18:10:23.438244+0000","last_peered":"2023-07-14T18:10:23.438244+0000","last_clean":"2023-07-14T18:10:23.438244+0000","last_became_active":"2023-07-14T18:10:22.452745+0000","last_became_peered":"2023-07-14T18:10:22.452745+0000","last_unstale":"2023-07-14T18:10:23.438244+0000","last_undegraded":"2023-07-14T18:10:23.438244+0000","last_fullsized":"2023-07-14T18:10:23.438244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:22:16.784956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438868+0000","last_change":"2023-07-14T18:10:17.215115+0000","last_active":"2023-07-14T18:10:23.438868+0000","last_peered":"2023-07-14T18:10:23.438868+0000","last_clean":"2023-07-14T18:10:23.438868+0000","last_became_active":"2023-07-14T18:10:17.214813+0000","last_became_peered":"2023-07-14T18:10:17.214813+0000","last_unstale":"2023-07-14T18:10:23.438868+0000","last_undegraded":"2023-07-14T18:10:23.438868+0000","last_fullsized":"2023-07-14T18:10:23.438868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:48:32.267436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090280+0000","last_change":"2023-07-14T18:10:18.119987+0000","last_active":"2023-07-14T18:10:24.090280+0000","last_peered":"2023-07-14T18:10:24.090280+0000","last_clean":"2023-07-14T18:10:24.090280+0000","last_became_active":"2023-07-14T18:10:18.119695+0000","last_became_peered":"2023-07-14T18:10:18.119695+0000","last_unstale":"2023-07-14T18:10:24.090280+0000","last_undegraded":"2023-07-14T18:10:24.090280+0000","last_fullsized":"2023-07-14T18:10:24.090280+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:46.295719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550007+0000","last_change":"2023-07-14T18:10:22.449689+0000","last_active":"2023-07-14T18:10:23.550007+0000","last_peered":"2023-07-14T18:10:23.550007+0000","last_clean":"2023-07-14T18:10:23.550007+0000","last_became_active":"2023-07-14T18:10:22.449273+0000","last_became_peered":"2023-07-14T18:10:22.449273+0000","last_unstale":"2023-07-14T18:10:23.550007+0000","last_undegraded":"2023-07-14T18:10:23.550007+0000","last_fullsized":"2023-07-14T18:10:23.550007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:54:23.642287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.982902+0000","last_change":"2023-07-14T18:10:17.216438+0000","last_active":"2023-07-14T18:10:36.982902+0000","last_peered":"2023-07-14T18:10:36.982902+0000","last_clean":"2023-07-14T18:10:36.982902+0000","last_became_active":"2023-07-14T18:10:17.216169+0000","last_became_peered":"2023-07-14T18:10:17.216169+0000","last_unstale":"2023-07-14T18:10:36.982902+0000","last_undegraded":"2023-07-14T18:10:36.982902+0000","last_fullsized":"2023-07-14T18:10:36.982902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:39.977001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438762+0000","last_change":"2023-07-14T18:10:18.112917+0000","last_active":"2023-07-14T18:10:23.438762+0000","last_peered":"2023-07-14T18:10:23.438762+0000","last_clean":"2023-07-14T18:10:23.438762+0000","last_became_active":"2023-07-14T18:10:18.112631+0000","last_became_peered":"2023-07-14T18:10:18.112631+0000","last_unstale":"2023-07-14T18:10:23.438762+0000","last_undegraded":"2023-07-14T18:10:23.438762+0000","last_fullsized":"2023-07-14T18:10:23.438762+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:14:01.774309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549936+0000","last_change":"2023-07-14T18:10:22.454278+0000","last_active":"2023-07-14T18:10:23.549936+0000","last_peered":"2023-07-14T18:10:23.549936+0000","last_clean":"2023-07-14T18:10:23.549936+0000","last_became_active":"2023-07-14T18:10:22.453834+0000","last_became_peered":"2023-07-14T18:10:22.453834+0000","last_unstale":"2023-07-14T18:10:23.549936+0000","last_undegraded":"2023-07-14T18:10:23.549936+0000","last_fullsized":"2023-07-14T18:10:23.549936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:31:24.983615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734643+0000","last_change":"2023-07-14T18:10:17.210217+0000","last_active":"2023-07-14T18:10:23.734643+0000","last_peered":"2023-07-14T18:10:23.734643+0000","last_clean":"2023-07-14T18:10:23.734643+0000","last_became_active":"2023-07-14T18:10:17.209953+0000","last_became_peered":"2023-07-14T18:10:17.209953+0000","last_unstale":"2023-07-14T18:10:23.734643+0000","last_undegraded":"2023-07-14T18:10:23.734643+0000","last_fullsized":"2023-07-14T18:10:23.734643+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:19:46.091265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090402+0000","last_change":"2023-07-14T18:10:18.113009+0000","last_active":"2023-07-14T18:10:24.090402+0000","last_peered":"2023-07-14T18:10:24.090402+0000","last_clean":"2023-07-14T18:10:24.090402+0000","last_became_active":"2023-07-14T18:10:18.112695+0000","last_became_peered":"2023-07-14T18:10:18.112695+0000","last_unstale":"2023-07-14T18:10:24.090402+0000","last_undegraded":"2023-07-14T18:10:24.090402+0000","last_fullsized":"2023-07-14T18:10:24.090402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:06:04.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089159+0000","last_change":"2023-07-14T18:10:22.457687+0000","last_active":"2023-07-14T18:10:24.089159+0000","last_peered":"2023-07-14T18:10:24.089159+0000","last_clean":"2023-07-14T18:10:24.089159+0000","last_became_active":"2023-07-14T18:10:22.457270+0000","last_became_peered":"2023-07-14T18:10:22.457270+0000","last_unstale":"2023-07-14T18:10:24.089159+0000","last_undegraded":"2023-07-14T18:10:24.089159+0000","last_fullsized":"2023-07-14T18:10:24.089159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:37:58.192527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429469+0000","last_change":"2023-07-14T18:10:17.209638+0000","last_active":"2023-07-14T18:10:37.429469+0000","last_peered":"2023-07-14T18:10:37.429469+0000","last_clean":"2023-07-14T18:10:37.429469+0000","last_became_active":"2023-07-14T18:10:17.209409+0000","last_became_peered":"2023-07-14T18:10:17.209409+0000","last_unstale":"2023-07-14T18:10:37.429469+0000","last_undegraded":"2023-07-14T18:10:37.429469+0000","last_fullsized":"2023-07-14T18:10:37.429469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:32.419102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088533+0000","last_change":"2023-07-14T18:10:18.109479+0000","last_active":"2023-07-14T18:10:24.088533+0000","last_peered":"2023-07-14T18:10:24.088533+0000","last_clean":"2023-07-14T18:10:24.088533+0000","last_became_active":"2023-07-14T18:10:18.109161+0000","last_became_peered":"2023-07-14T18:10:18.109161+0000","last_unstale":"2023-07-14T18:10:24.088533+0000","last_undegraded":"2023-07-14T18:10:24.088533+0000","last_fullsized":"2023-07-14T18:10:24.088533+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:05.204295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440596+0000","last_change":"2023-07-14T18:10:22.453731+0000","last_active":"2023-07-14T18:10:23.440596+0000","last_peered":"2023-07-14T18:10:23.440596+0000","last_clean":"2023-07-14T18:10:23.440596+0000","last_became_active":"2023-07-14T18:10:22.453311+0000","last_became_peered":"2023-07-14T18:10:22.453311+0000","last_unstale":"2023-07-14T18:10:23.440596+0000","last_undegraded":"2023-07-14T18:10:23.440596+0000","last_fullsized":"2023-07-14T18:10:23.440596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:35.892698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089898+0000","last_change":"2023-07-14T18:10:17.218050+0000","last_active":"2023-07-14T18:10:24.089898+0000","last_peered":"2023-07-14T18:10:24.089898+0000","last_clean":"2023-07-14T18:10:24.089898+0000","last_became_active":"2023-07-14T18:10:17.217793+0000","last_became_peered":"2023-07-14T18:10:17.217793+0000","last_unstale":"2023-07-14T18:10:24.089898+0000","last_undegraded":"2023-07-14T18:10:24.089898+0000","last_fullsized":"2023-07-14T18:10:24.089898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:23.781098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089467+0000","last_change":"2023-07-14T18:10:18.111598+0000","last_active":"2023-07-14T18:10:24.089467+0000","last_peered":"2023-07-14T18:10:24.089467+0000","last_clean":"2023-07-14T18:10:24.089467+0000","last_became_active":"2023-07-14T18:10:18.111244+0000","last_became_peered":"2023-07-14T18:10:18.111244+0000","last_unstale":"2023-07-14T18:10:24.089467+0000","last_undegraded":"2023-07-14T18:10:24.089467+0000","last_fullsized":"2023-07-14T18:10:24.089467+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:26:47.187587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088793+0000","last_change":"2023-07-14T18:10:22.447676+0000","last_active":"2023-07-14T18:10:24.088793+0000","last_peered":"2023-07-14T18:10:24.088793+0000","last_clean":"2023-07-14T18:10:24.088793+0000","last_became_active":"2023-07-14T18:10:22.446648+0000","last_became_peered":"2023-07-14T18:10:22.446648+0000","last_unstale":"2023-07-14T18:10:24.088793+0000","last_undegraded":"2023-07-14T18:10:24.088793+0000","last_fullsized":"2023-07-14T18:10:24.088793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:17:30.941899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.735022+0000","last_change":"2023-07-14T18:10:17.213897+0000","last_active":"2023-07-14T18:10:23.735022+0000","last_peered":"2023-07-14T18:10:23.735022+0000","last_clean":"2023-07-14T18:10:23.735022+0000","last_became_active":"2023-07-14T18:10:17.213547+0000","last_became_peered":"2023-07-14T18:10:17.213547+0000","last_unstale":"2023-07-14T18:10:23.735022+0000","last_undegraded":"2023-07-14T18:10:23.735022+0000","last_fullsized":"2023-07-14T18:10:23.735022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:44.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089327+0000","last_change":"2023-07-14T18:10:18.114776+0000","last_active":"2023-07-14T18:10:24.089327+0000","last_peered":"2023-07-14T18:10:24.089327+0000","last_clean":"2023-07-14T18:10:24.089327+0000","last_became_active":"2023-07-14T18:10:18.112469+0000","last_became_peered":"2023-07-14T18:10:18.112469+0000","last_unstale":"2023-07-14T18:10:24.089327+0000","last_undegraded":"2023-07-14T18:10:24.089327+0000","last_fullsized":"2023-07-14T18:10:24.089327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:30.864496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088873+0000","last_change":"2023-07-14T18:10:22.454215+0000","last_active":"2023-07-14T18:10:24.088873+0000","last_peered":"2023-07-14T18:10:24.088873+0000","last_clean":"2023-07-14T18:10:24.088873+0000","last_became_active":"2023-07-14T18:10:22.453889+0000","last_became_peered":"2023-07-14T18:10:22.453889+0000","last_unstale":"2023-07-14T18:10:24.088873+0000","last_undegraded":"2023-07-14T18:10:24.088873+0000","last_fullsized":"2023-07-14T18:10:24.088873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:30:09.159482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981368+0000","last_change":"2023-07-14T18:10:17.212602+0000","last_active":"2023-07-14T18:10:36.981368+0000","last_peered":"2023-07-14T18:10:36.981368+0000","last_clean":"2023-07-14T18:10:36.981368+0000","last_became_active":"2023-07-14T18:10:17.212087+0000","last_became_peered":"2023-07-14T18:10:17.212087+0000","last_unstale":"2023-07-14T18:10:36.981368+0000","last_undegraded":"2023-07-14T18:10:36.981368+0000","last_fullsized":"2023-07-14T18:10:36.981368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:30:00.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440688+0000","last_change":"2023-07-14T18:10:18.116972+0000","last_active":"2023-07-14T18:10:23.440688+0000","last_peered":"2023-07-14T18:10:23.440688+0000","last_clean":"2023-07-14T18:10:23.440688+0000","last_became_active":"2023-07-14T18:10:18.116423+0000","last_became_peered":"2023-07-14T18:10:18.116423+0000","last_unstale":"2023-07-14T18:10:23.440688+0000","last_undegraded":"2023-07-14T18:10:23.440688+0000","last_fullsized":"2023-07-14T18:10:23.440688+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:53.315535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438563+0000","last_change":"2023-07-14T18:10:22.450207+0000","last_active":"2023-07-14T18:10:23.438563+0000","last_peered":"2023-07-14T18:10:23.438563+0000","last_clean":"2023-07-14T18:10:23.438563+0000","last_became_active":"2023-07-14T18:10:22.449826+0000","last_became_peered":"2023-07-14T18:10:22.449826+0000","last_unstale":"2023-07-14T18:10:23.438563+0000","last_undegraded":"2023-07-14T18:10:23.438563+0000","last_fullsized":"2023-07-14T18:10:23.438563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:18.013945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090602+0000","last_change":"2023-07-14T18:10:17.213769+0000","last_active":"2023-07-14T18:10:24.090602+0000","last_peered":"2023-07-14T18:10:24.090602+0000","last_clean":"2023-07-14T18:10:24.090602+0000","last_became_active":"2023-07-14T18:10:17.213465+0000","last_became_peered":"2023-07-14T18:10:17.213465+0000","last_unstale":"2023-07-14T18:10:24.090602+0000","last_undegraded":"2023-07-14T18:10:24.090602+0000","last_fullsized":"2023-07-14T18:10:24.090602+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:38:07.890716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089153+0000","last_change":"2023-07-14T18:10:18.122764+0000","last_active":"2023-07-14T18:10:24.089153+0000","last_peered":"2023-07-14T18:10:24.089153+0000","last_clean":"2023-07-14T18:10:24.089153+0000","last_became_active":"2023-07-14T18:10:18.120522+0000","last_became_peered":"2023-07-14T18:10:18.120522+0000","last_unstale":"2023-07-14T18:10:24.089153+0000","last_undegraded":"2023-07-14T18:10:24.089153+0000","last_fullsized":"2023-07-14T18:10:24.089153+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:54:07.567008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089725+0000","last_change":"2023-07-14T18:10:22.455635+0000","last_active":"2023-07-14T18:10:24.089725+0000","last_peered":"2023-07-14T18:10:24.089725+0000","last_clean":"2023-07-14T18:10:24.089725+0000","last_became_active":"2023-07-14T18:10:22.455128+0000","last_became_peered":"2023-07-14T18:10:22.455128+0000","last_unstale":"2023-07-14T18:10:24.089725+0000","last_undegraded":"2023-07-14T18:10:24.089725+0000","last_fullsized":"2023-07-14T18:10:24.089725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:01.364723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'10","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:34.473808+0000","last_change":"2023-07-14T18:10:17.215255+0000","last_active":"2023-07-14T18:10:34.473808+0000","last_peered":"2023-07-14T18:10:34.473808+0000","last_clean":"2023-07-14T18:10:34.473808+0000","last_became_active":"2023-07-14T18:10:17.214900+0000","last_became_peered":"2023-07-14T18:10:17.214900+0000","last_unstale":"2023-07-14T18:10:34.473808+0000","last_undegraded":"2023-07-14T18:10:34.473808+0000","last_fullsized":"2023-07-14T18:10:34.473808+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:01.480644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438902+0000","last_change":"2023-07-14T18:10:18.115355+0000","last_active":"2023-07-14T18:10:23.438902+0000","last_peered":"2023-07-14T18:10:23.438902+0000","last_clean":"2023-07-14T18:10:23.438902+0000","last_became_active":"2023-07-14T18:10:18.115023+0000","last_became_peered":"2023-07-14T18:10:18.115023+0000","last_unstale":"2023-07-14T18:10:23.438902+0000","last_undegraded":"2023-07-14T18:10:23.438902+0000","last_fullsized":"2023-07-14T18:10:23.438902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:16.057123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090803+0000","last_change":"2023-07-14T18:10:22.453113+0000","last_active":"2023-07-14T18:10:24.090803+0000","last_peered":"2023-07-14T18:10:24.090803+0000","last_clean":"2023-07-14T18:10:24.090803+0000","last_became_active":"2023-07-14T18:10:22.452688+0000","last_became_peered":"2023-07-14T18:10:22.452688+0000","last_unstale":"2023-07-14T18:10:24.090803+0000","last_undegraded":"2023-07-14T18:10:24.090803+0000","last_fullsized":"2023-07-14T18:10:24.090803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:47.829022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430020+0000","last_change":"2023-07-14T18:10:17.210924+0000","last_active":"2023-07-14T18:10:37.430020+0000","last_peered":"2023-07-14T18:10:37.430020+0000","last_clean":"2023-07-14T18:10:37.430020+0000","last_became_active":"2023-07-14T18:10:17.210604+0000","last_became_peered":"2023-07-14T18:10:17.210604+0000","last_unstale":"2023-07-14T18:10:37.430020+0000","last_undegraded":"2023-07-14T18:10:37.430020+0000","last_fullsized":"2023-07-14T18:10:37.430020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:10:28.635558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845862+0000","last_change":"2023-07-14T18:10:18.115094+0000","last_active":"2023-07-14T18:10:23.845862+0000","last_peered":"2023-07-14T18:10:23.845862+0000","last_clean":"2023-07-14T18:10:23.845862+0000","last_became_active":"2023-07-14T18:10:18.114538+0000","last_became_peered":"2023-07-14T18:10:18.114538+0000","last_unstale":"2023-07-14T18:10:23.845862+0000","last_undegraded":"2023-07-14T18:10:23.845862+0000","last_fullsized":"2023-07-14T18:10:23.845862+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:19:15.916223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088869+0000","last_change":"2023-07-14T18:10:22.459151+0000","last_active":"2023-07-14T18:10:24.088869+0000","last_peered":"2023-07-14T18:10:24.088869+0000","last_clean":"2023-07-14T18:10:24.088869+0000","last_became_active":"2023-07-14T18:10:22.458550+0000","last_became_peered":"2023-07-14T18:10:22.458550+0000","last_unstale":"2023-07-14T18:10:24.088869+0000","last_undegraded":"2023-07-14T18:10:24.088869+0000","last_fullsized":"2023-07-14T18:10:24.088869+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:11.750342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.982150+0000","last_change":"2023-07-14T18:10:17.216475+0000","last_active":"2023-07-14T18:10:36.982150+0000","last_peered":"2023-07-14T18:10:36.982150+0000","last_clean":"2023-07-14T18:10:36.982150+0000","last_became_active":"2023-07-14T18:10:17.216148+0000","last_became_peered":"2023-07-14T18:10:17.216148+0000","last_unstale":"2023-07-14T18:10:36.982150+0000","last_undegraded":"2023-07-14T18:10:36.982150+0000","last_fullsized":"2023-07-14T18:10:36.982150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:00:50.106730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089545+0000","last_change":"2023-07-14T18:10:18.121764+0000","last_active":"2023-07-14T18:10:24.089545+0000","last_peered":"2023-07-14T18:10:24.089545+0000","last_clean":"2023-07-14T18:10:24.089545+0000","last_became_active":"2023-07-14T18:10:18.121458+0000","last_became_peered":"2023-07-14T18:10:18.121458+0000","last_unstale":"2023-07-14T18:10:24.089545+0000","last_undegraded":"2023-07-14T18:10:24.089545+0000","last_fullsized":"2023-07-14T18:10:24.089545+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:26:22.145994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089867+0000","last_change":"2023-07-14T18:10:22.457737+0000","last_active":"2023-07-14T18:10:24.089867+0000","last_peered":"2023-07-14T18:10:24.089867+0000","last_clean":"2023-07-14T18:10:24.089867+0000","last_became_active":"2023-07-14T18:10:22.457359+0000","last_became_peered":"2023-07-14T18:10:22.457359+0000","last_unstale":"2023-07-14T18:10:24.089867+0000","last_undegraded":"2023-07-14T18:10:24.089867+0000","last_fullsized":"2023-07-14T18:10:24.089867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:55.574003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438236+0000","last_change":"2023-07-14T18:10:17.214663+0000","last_active":"2023-07-14T18:10:23.438236+0000","last_peered":"2023-07-14T18:10:23.438236+0000","last_clean":"2023-07-14T18:10:23.438236+0000","last_became_active":"2023-07-14T18:10:17.214355+0000","last_became_peered":"2023-07-14T18:10:17.214355+0000","last_unstale":"2023-07-14T18:10:23.438236+0000","last_undegraded":"2023-07-14T18:10:23.438236+0000","last_fullsized":"2023-07-14T18:10:23.438236+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:56.306784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088750+0000","last_change":"2023-07-14T18:10:18.124903+0000","last_active":"2023-07-14T18:10:24.088750+0000","last_peered":"2023-07-14T18:10:24.088750+0000","last_clean":"2023-07-14T18:10:24.088750+0000","last_became_active":"2023-07-14T18:10:18.124661+0000","last_became_peered":"2023-07-14T18:10:18.124661+0000","last_unstale":"2023-07-14T18:10:24.088750+0000","last_undegraded":"2023-07-14T18:10:24.088750+0000","last_fullsized":"2023-07-14T18:10:24.088750+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:21.959726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846209+0000","last_change":"2023-07-14T18:10:22.459746+0000","last_active":"2023-07-14T18:10:23.846209+0000","last_peered":"2023-07-14T18:10:23.846209+0000","last_clean":"2023-07-14T18:10:23.846209+0000","last_became_active":"2023-07-14T18:10:22.459331+0000","last_became_peered":"2023-07-14T18:10:22.459331+0000","last_unstale":"2023-07-14T18:10:23.846209+0000","last_undegraded":"2023-07-14T18:10:23.846209+0000","last_fullsized":"2023-07-14T18:10:23.846209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:34.082947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430093+0000","last_change":"2023-07-14T18:10:17.213362+0000","last_active":"2023-07-14T18:10:37.430093+0000","last_peered":"2023-07-14T18:10:37.430093+0000","last_clean":"2023-07-14T18:10:37.430093+0000","last_became_active":"2023-07-14T18:10:17.212914+0000","last_became_peered":"2023-07-14T18:10:17.212914+0000","last_unstale":"2023-07-14T18:10:37.430093+0000","last_undegraded":"2023-07-14T18:10:37.430093+0000","last_fullsized":"2023-07-14T18:10:37.430093+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:08.051946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090127+0000","last_change":"2023-07-14T18:10:18.106583+0000","last_active":"2023-07-14T18:10:24.090127+0000","last_peered":"2023-07-14T18:10:24.090127+0000","last_clean":"2023-07-14T18:10:24.090127+0000","last_became_active":"2023-07-14T18:10:18.106216+0000","last_became_peered":"2023-07-14T18:10:18.106216+0000","last_unstale":"2023-07-14T18:10:24.090127+0000","last_undegraded":"2023-07-14T18:10:24.090127+0000","last_fullsized":"2023-07-14T18:10:24.090127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:03:32.095853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089940+0000","last_change":"2023-07-14T18:10:22.447133+0000","last_active":"2023-07-14T18:10:24.089940+0000","last_peered":"2023-07-14T18:10:24.089940+0000","last_clean":"2023-07-14T18:10:24.089940+0000","last_became_active":"2023-07-14T18:10:22.446716+0000","last_became_peered":"2023-07-14T18:10:22.446716+0000","last_unstale":"2023-07-14T18:10:24.089940+0000","last_undegraded":"2023-07-14T18:10:24.089940+0000","last_fullsized":"2023-07-14T18:10:24.089940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:49.677906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.439030+0000","last_change":"2023-07-14T18:10:17.210729+0000","last_active":"2023-07-14T18:10:23.439030+0000","last_peered":"2023-07-14T18:10:23.439030+0000","last_clean":"2023-07-14T18:10:23.439030+0000","last_became_active":"2023-07-14T18:10:17.210449+0000","last_became_peered":"2023-07-14T18:10:17.210449+0000","last_unstale":"2023-07-14T18:10:23.439030+0000","last_undegraded":"2023-07-14T18:10:23.439030+0000","last_fullsized":"2023-07-14T18:10:23.439030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:57.909641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089476+0000","last_change":"2023-07-14T18:10:18.107118+0000","last_active":"2023-07-14T18:10:24.089476+0000","last_peered":"2023-07-14T18:10:24.089476+0000","last_clean":"2023-07-14T18:10:24.089476+0000","last_became_active":"2023-07-14T18:10:18.106367+0000","last_became_peered":"2023-07-14T18:10:18.106367+0000","last_unstale":"2023-07-14T18:10:24.089476+0000","last_undegraded":"2023-07-14T18:10:24.089476+0000","last_fullsized":"2023-07-14T18:10:24.089476+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:50:19.299272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438592+0000","last_change":"2023-07-14T18:10:22.500692+0000","last_active":"2023-07-14T18:10:23.438592+0000","last_peered":"2023-07-14T18:10:23.438592+0000","last_clean":"2023-07-14T18:10:23.438592+0000","last_became_active":"2023-07-14T18:10:22.500239+0000","last_became_peered":"2023-07-14T18:10:22.500239+0000","last_unstale":"2023-07-14T18:10:23.438592+0000","last_undegraded":"2023-07-14T18:10:23.438592+0000","last_fullsized":"2023-07-14T18:10:23.438592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:06.716562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089450+0000","last_change":"2023-07-14T18:10:17.212519+0000","last_active":"2023-07-14T18:10:24.089450+0000","last_peered":"2023-07-14T18:10:24.089450+0000","last_clean":"2023-07-14T18:10:24.089450+0000","last_became_active":"2023-07-14T18:10:17.212228+0000","last_became_peered":"2023-07-14T18:10:17.212228+0000","last_unstale":"2023-07-14T18:10:24.089450+0000","last_undegraded":"2023-07-14T18:10:24.089450+0000","last_fullsized":"2023-07-14T18:10:24.089450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:25:19.995310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734905+0000","last_change":"2023-07-14T18:10:18.111808+0000","last_active":"2023-07-14T18:10:23.734905+0000","last_peered":"2023-07-14T18:10:23.734905+0000","last_clean":"2023-07-14T18:10:23.734905+0000","last_became_active":"2023-07-14T18:10:18.111501+0000","last_became_peered":"2023-07-14T18:10:18.111501+0000","last_unstale":"2023-07-14T18:10:23.734905+0000","last_undegraded":"2023-07-14T18:10:23.734905+0000","last_fullsized":"2023-07-14T18:10:23.734905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:36.111363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845939+0000","last_change":"2023-07-14T18:10:22.453153+0000","last_active":"2023-07-14T18:10:23.845939+0000","last_peered":"2023-07-14T18:10:23.845939+0000","last_clean":"2023-07-14T18:10:23.845939+0000","last_became_active":"2023-07-14T18:10:22.452767+0000","last_became_peered":"2023-07-14T18:10:22.452767+0000","last_unstale":"2023-07-14T18:10:23.845939+0000","last_undegraded":"2023-07-14T18:10:23.845939+0000","last_fullsized":"2023-07-14T18:10:23.845939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:10:07.917749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089769+0000","last_change":"2023-07-14T18:10:17.215950+0000","last_active":"2023-07-14T18:10:24.089769+0000","last_peered":"2023-07-14T18:10:24.089769+0000","last_clean":"2023-07-14T18:10:24.089769+0000","last_became_active":"2023-07-14T18:10:17.215690+0000","last_became_peered":"2023-07-14T18:10:17.215690+0000","last_unstale":"2023-07-14T18:10:24.089769+0000","last_undegraded":"2023-07-14T18:10:24.089769+0000","last_fullsized":"2023-07-14T18:10:24.089769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:54.597911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734912+0000","last_change":"2023-07-14T18:10:18.114043+0000","last_active":"2023-07-14T18:10:23.734912+0000","last_peered":"2023-07-14T18:10:23.734912+0000","last_clean":"2023-07-14T18:10:23.734912+0000","last_became_active":"2023-07-14T18:10:18.113745+0000","last_became_peered":"2023-07-14T18:10:18.113745+0000","last_unstale":"2023-07-14T18:10:23.734912+0000","last_undegraded":"2023-07-14T18:10:23.734912+0000","last_fullsized":"2023-07-14T18:10:23.734912+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:42:18.691206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846054+0000","last_change":"2023-07-14T18:10:22.501904+0000","last_active":"2023-07-14T18:10:23.846054+0000","last_peered":"2023-07-14T18:10:23.846054+0000","last_clean":"2023-07-14T18:10:23.846054+0000","last_became_active":"2023-07-14T18:10:22.501500+0000","last_became_peered":"2023-07-14T18:10:22.501500+0000","last_unstale":"2023-07-14T18:10:23.846054+0000","last_undegraded":"2023-07-14T18:10:23.846054+0000","last_fullsized":"2023-07-14T18:10:23.846054+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:27:26.164775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090043+0000","last_change":"2023-07-14T18:10:18.114655+0000","last_active":"2023-07-14T18:10:24.090043+0000","last_peered":"2023-07-14T18:10:24.090043+0000","last_clean":"2023-07-14T18:10:24.090043+0000","last_became_active":"2023-07-14T18:10:18.113984+0000","last_became_peered":"2023-07-14T18:10:18.113984+0000","last_unstale":"2023-07-14T18:10:24.090043+0000","last_undegraded":"2023-07-14T18:10:24.090043+0000","last_fullsized":"2023-07-14T18:10:24.090043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:51:36.108289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430903+0000","last_change":"2023-07-14T18:10:17.212612+0000","last_active":"2023-07-14T18:10:37.430903+0000","last_peered":"2023-07-14T18:10:37.430903+0000","last_clean":"2023-07-14T18:10:37.430903+0000","last_became_active":"2023-07-14T18:10:17.212282+0000","last_became_peered":"2023-07-14T18:10:17.212282+0000","last_unstale":"2023-07-14T18:10:37.430903+0000","last_undegraded":"2023-07-14T18:10:37.430903+0000","last_fullsized":"2023-07-14T18:10:37.430903+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:12:46.994247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089078+0000","last_change":"2023-07-14T18:10:22.449831+0000","last_active":"2023-07-14T18:10:24.089078+0000","last_peered":"2023-07-14T18:10:24.089078+0000","last_clean":"2023-07-14T18:10:24.089078+0000","last_became_active":"2023-07-14T18:10:22.449380+0000","last_became_peered":"2023-07-14T18:10:22.449380+0000","last_unstale":"2023-07-14T18:10:24.089078+0000","last_undegraded":"2023-07-14T18:10:24.089078+0000","last_fullsized":"2023-07-14T18:10:24.089078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:33.180987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088819+0000","last_change":"2023-07-14T18:10:18.107041+0000","last_active":"2023-07-14T18:10:24.088819+0000","last_peered":"2023-07-14T18:10:24.088819+0000","last_clean":"2023-07-14T18:10:24.088819+0000","last_became_active":"2023-07-14T18:10:18.106715+0000","last_became_peered":"2023-07-14T18:10:18.106715+0000","last_unstale":"2023-07-14T18:10:24.088819+0000","last_undegraded":"2023-07-14T18:10:24.088819+0000","last_fullsized":"2023-07-14T18:10:24.088819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:49.905307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438471+0000","last_change":"2023-07-14T18:10:17.205525+0000","last_active":"2023-07-14T18:10:23.438471+0000","last_peered":"2023-07-14T18:10:23.438471+0000","last_clean":"2023-07-14T18:10:23.438471+0000","last_became_active":"2023-07-14T18:10:17.205164+0000","last_became_peered":"2023-07-14T18:10:17.205164+0000","last_unstale":"2023-07-14T18:10:23.438471+0000","last_undegraded":"2023-07-14T18:10:23.438471+0000","last_fullsized":"2023-07-14T18:10:23.438471+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:06:06.720104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088564+0000","last_change":"2023-07-14T18:09:18.495776+0000","last_active":"2023-07-14T18:10:24.088564+0000","last_peered":"2023-07-14T18:10:24.088564+0000","last_clean":"2023-07-14T18:10:24.088564+0000","last_became_active":"2023-07-14T18:09:18.495326+0000","last_became_peered":"2023-07-14T18:09:18.495326+0000","last_unstale":"2023-07-14T18:10:24.088564+0000","last_undegraded":"2023-07-14T18:10:24.088564+0000","last_fullsized":"2023-07-14T18:10:24.088564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:06:31.483975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:06:31.483975+0000","last_clean_scrub_stamp":"2023-07-14T18:06:31.483975+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:42:18.539340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090230+0000","last_change":"2023-07-14T18:10:22.501067+0000","last_active":"2023-07-14T18:10:24.090230+0000","last_peered":"2023-07-14T18:10:24.090230+0000","last_clean":"2023-07-14T18:10:24.090230+0000","last_became_active":"2023-07-14T18:10:22.500337+0000","last_became_peered":"2023-07-14T18:10:22.500337+0000","last_unstale":"2023-07-14T18:10:24.090230+0000","last_undegraded":"2023-07-14T18:10:24.090230+0000","last_fullsized":"2023-07-14T18:10:24.090230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:43.900787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845551+0000","last_change":"2023-07-14T18:10:18.107299+0000","last_active":"2023-07-14T18:10:23.845551+0000","last_peered":"2023-07-14T18:10:23.845551+0000","last_clean":"2023-07-14T18:10:23.845551+0000","last_became_active":"2023-07-14T18:10:18.106986+0000","last_became_peered":"2023-07-14T18:10:18.106986+0000","last_unstale":"2023-07-14T18:10:23.845551+0000","last_undegraded":"2023-07-14T18:10:23.845551+0000","last_fullsized":"2023-07-14T18:10:23.845551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:47.211264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429208+0000","last_change":"2023-07-14T18:10:17.217895+0000","last_active":"2023-07-14T18:10:37.429208+0000","last_peered":"2023-07-14T18:10:37.429208+0000","last_clean":"2023-07-14T18:10:37.429208+0000","last_became_active":"2023-07-14T18:10:17.217548+0000","last_became_peered":"2023-07-14T18:10:17.217548+0000","last_unstale":"2023-07-14T18:10:37.429208+0000","last_undegraded":"2023-07-14T18:10:37.429208+0000","last_fullsized":"2023-07-14T18:10:37.429208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:29:08.392282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088848+0000","last_change":"2023-07-14T18:10:22.453140+0000","last_active":"2023-07-14T18:10:24.088848+0000","last_peered":"2023-07-14T18:10:24.088848+0000","last_clean":"2023-07-14T18:10:24.088848+0000","last_became_active":"2023-07-14T18:10:22.452731+0000","last_became_peered":"2023-07-14T18:10:22.452731+0000","last_unstale":"2023-07-14T18:10:24.088848+0000","last_undegraded":"2023-07-14T18:10:24.088848+0000","last_fullsized":"2023-07-14T18:10:24.088848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:40.973261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980328+0000","last_change":"2023-07-14T18:10:17.213431+0000","last_active":"2023-07-14T18:10:36.980328+0000","last_peered":"2023-07-14T18:10:36.980328+0000","last_clean":"2023-07-14T18:10:36.980328+0000","last_became_active":"2023-07-14T18:10:17.213125+0000","last_became_peered":"2023-07-14T18:10:17.213125+0000","last_unstale":"2023-07-14T18:10:36.980328+0000","last_undegraded":"2023-07-14T18:10:36.980328+0000","last_fullsized":"2023-07-14T18:10:36.980328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:53:36.345583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846032+0000","last_change":"2023-07-14T18:10:18.110593+0000","last_active":"2023-07-14T18:10:23.846032+0000","last_peered":"2023-07-14T18:10:23.846032+0000","last_clean":"2023-07-14T18:10:23.846032+0000","last_became_active":"2023-07-14T18:10:18.110220+0000","last_became_peered":"2023-07-14T18:10:18.110220+0000","last_unstale":"2023-07-14T18:10:23.846032+0000","last_undegraded":"2023-07-14T18:10:23.846032+0000","last_fullsized":"2023-07-14T18:10:23.846032+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:58:18.893324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088540+0000","last_change":"2023-07-14T18:10:22.500809+0000","last_active":"2023-07-14T18:10:24.088540+0000","last_peered":"2023-07-14T18:10:24.088540+0000","last_clean":"2023-07-14T18:10:24.088540+0000","last_became_active":"2023-07-14T18:10:22.500387+0000","last_became_peered":"2023-07-14T18:10:22.500387+0000","last_unstale":"2023-07-14T18:10:24.088540+0000","last_undegraded":"2023-07-14T18:10:24.088540+0000","last_fullsized":"2023-07-14T18:10:24.088540+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:26.241462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089871+0000","last_change":"2023-07-14T18:10:17.209632+0000","last_active":"2023-07-14T18:10:24.089871+0000","last_peered":"2023-07-14T18:10:24.089871+0000","last_clean":"2023-07-14T18:10:24.089871+0000","last_became_active":"2023-07-14T18:10:17.209274+0000","last_became_peered":"2023-07-14T18:10:17.209274+0000","last_unstale":"2023-07-14T18:10:24.089871+0000","last_undegraded":"2023-07-14T18:10:24.089871+0000","last_fullsized":"2023-07-14T18:10:24.089871+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:49:02.203177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089197+0000","last_change":"2023-07-14T18:10:18.119828+0000","last_active":"2023-07-14T18:10:24.089197+0000","last_peered":"2023-07-14T18:10:24.089197+0000","last_clean":"2023-07-14T18:10:24.089197+0000","last_became_active":"2023-07-14T18:10:18.119551+0000","last_became_peered":"2023-07-14T18:10:18.119551+0000","last_unstale":"2023-07-14T18:10:24.089197+0000","last_undegraded":"2023-07-14T18:10:24.089197+0000","last_fullsized":"2023-07-14T18:10:24.089197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:59:11.985338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549754+0000","last_change":"2023-07-14T18:10:22.450325+0000","last_active":"2023-07-14T18:10:23.549754+0000","last_peered":"2023-07-14T18:10:23.549754+0000","last_clean":"2023-07-14T18:10:23.549754+0000","last_became_active":"2023-07-14T18:10:22.449492+0000","last_became_peered":"2023-07-14T18:10:22.449492+0000","last_unstale":"2023-07-14T18:10:23.549754+0000","last_undegraded":"2023-07-14T18:10:23.549754+0000","last_fullsized":"2023-07-14T18:10:23.549754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:37:37.829472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438842+0000","last_change":"2023-07-14T18:10:17.212891+0000","last_active":"2023-07-14T18:10:23.438842+0000","last_peered":"2023-07-14T18:10:23.438842+0000","last_clean":"2023-07-14T18:10:23.438842+0000","last_became_active":"2023-07-14T18:10:17.212573+0000","last_became_peered":"2023-07-14T18:10:17.212573+0000","last_unstale":"2023-07-14T18:10:23.438842+0000","last_undegraded":"2023-07-14T18:10:23.438842+0000","last_fullsized":"2023-07-14T18:10:23.438842+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:38:07.247348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440729+0000","last_change":"2023-07-14T18:10:18.113389+0000","last_active":"2023-07-14T18:10:23.440729+0000","last_peered":"2023-07-14T18:10:23.440729+0000","last_clean":"2023-07-14T18:10:23.440729+0000","last_became_active":"2023-07-14T18:10:18.113053+0000","last_became_peered":"2023-07-14T18:10:18.113053+0000","last_unstale":"2023-07-14T18:10:23.440729+0000","last_undegraded":"2023-07-14T18:10:23.440729+0000","last_fullsized":"2023-07-14T18:10:23.440729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:29:31.077735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438597+0000","last_change":"2023-07-14T18:10:22.500749+0000","last_active":"2023-07-14T18:10:23.438597+0000","last_peered":"2023-07-14T18:10:23.438597+0000","last_clean":"2023-07-14T18:10:23.438597+0000","last_became_active":"2023-07-14T18:10:22.500357+0000","last_became_peered":"2023-07-14T18:10:22.500357+0000","last_unstale":"2023-07-14T18:10:23.438597+0000","last_undegraded":"2023-07-14T18:10:23.438597+0000","last_fullsized":"2023-07-14T18:10:23.438597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:05:10.947498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980089+0000","last_change":"2023-07-14T18:10:17.212501+0000","last_active":"2023-07-14T18:10:36.980089+0000","last_peered":"2023-07-14T18:10:36.980089+0000","last_clean":"2023-07-14T18:10:36.980089+0000","last_became_active":"2023-07-14T18:10:17.212171+0000","last_became_peered":"2023-07-14T18:10:17.212171+0000","last_unstale":"2023-07-14T18:10:36.980089+0000","last_undegraded":"2023-07-14T18:10:36.980089+0000","last_fullsized":"2023-07-14T18:10:36.980089+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:37.208644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438420+0000","last_change":"2023-07-14T18:10:18.111692+0000","last_active":"2023-07-14T18:10:23.438420+0000","last_peered":"2023-07-14T18:10:23.438420+0000","last_clean":"2023-07-14T18:10:23.438420+0000","last_became_active":"2023-07-14T18:10:18.111424+0000","last_became_peered":"2023-07-14T18:10:18.111424+0000","last_unstale":"2023-07-14T18:10:23.438420+0000","last_undegraded":"2023-07-14T18:10:23.438420+0000","last_fullsized":"2023-07-14T18:10:23.438420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:06.610911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089989+0000","last_change":"2023-07-14T18:10:22.456773+0000","last_active":"2023-07-14T18:10:24.089989+0000","last_peered":"2023-07-14T18:10:24.089989+0000","last_clean":"2023-07-14T18:10:24.089989+0000","last_became_active":"2023-07-14T18:10:22.456309+0000","last_became_peered":"2023-07-14T18:10:22.456309+0000","last_unstale":"2023-07-14T18:10:24.089989+0000","last_undegraded":"2023-07-14T18:10:24.089989+0000","last_fullsized":"2023-07-14T18:10:24.089989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:29:09.644386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088932+0000","last_change":"2023-07-14T18:10:17.205109+0000","last_active":"2023-07-14T18:10:24.088932+0000","last_peered":"2023-07-14T18:10:24.088932+0000","last_clean":"2023-07-14T18:10:24.088932+0000","last_became_active":"2023-07-14T18:10:17.204644+0000","last_became_peered":"2023-07-14T18:10:17.204644+0000","last_unstale":"2023-07-14T18:10:24.088932+0000","last_undegraded":"2023-07-14T18:10:24.088932+0000","last_fullsized":"2023-07-14T18:10:24.088932+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:08.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088692+0000","last_change":"2023-07-14T18:10:18.109143+0000","last_active":"2023-07-14T18:10:24.088692+0000","last_peered":"2023-07-14T18:10:24.088692+0000","last_clean":"2023-07-14T18:10:24.088692+0000","last_became_active":"2023-07-14T18:10:18.108841+0000","last_became_peered":"2023-07-14T18:10:18.108841+0000","last_unstale":"2023-07-14T18:10:24.088692+0000","last_undegraded":"2023-07-14T18:10:24.088692+0000","last_fullsized":"2023-07-14T18:10:24.088692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:30:16.212954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438316+0000","last_change":"2023-07-14T18:10:22.450564+0000","last_active":"2023-07-14T18:10:23.438316+0000","last_peered":"2023-07-14T18:10:23.438316+0000","last_clean":"2023-07-14T18:10:23.438316+0000","last_became_active":"2023-07-14T18:10:22.449801+0000","last_became_peered":"2023-07-14T18:10:22.449801+0000","last_unstale":"2023-07-14T18:10:23.438316+0000","last_undegraded":"2023-07-14T18:10:23.438316+0000","last_fullsized":"2023-07-14T18:10:23.438316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:00:04.711449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549788+0000","last_change":"2023-07-14T18:10:17.211522+0000","last_active":"2023-07-14T18:10:23.549788+0000","last_peered":"2023-07-14T18:10:23.549788+0000","last_clean":"2023-07-14T18:10:23.549788+0000","last_became_active":"2023-07-14T18:10:17.211163+0000","last_became_peered":"2023-07-14T18:10:17.211163+0000","last_unstale":"2023-07-14T18:10:23.549788+0000","last_undegraded":"2023-07-14T18:10:23.549788+0000","last_fullsized":"2023-07-14T18:10:23.549788+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:14:47.230105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089155+0000","last_change":"2023-07-14T18:10:18.111885+0000","last_active":"2023-07-14T18:10:24.089155+0000","last_peered":"2023-07-14T18:10:24.089155+0000","last_clean":"2023-07-14T18:10:24.089155+0000","last_became_active":"2023-07-14T18:10:18.111520+0000","last_became_peered":"2023-07-14T18:10:18.111520+0000","last_unstale":"2023-07-14T18:10:24.089155+0000","last_undegraded":"2023-07-14T18:10:24.089155+0000","last_fullsized":"2023-07-14T18:10:24.089155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:06:29.717829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438617+0000","last_change":"2023-07-14T18:10:22.500612+0000","last_active":"2023-07-14T18:10:23.438617+0000","last_peered":"2023-07-14T18:10:23.438617+0000","last_clean":"2023-07-14T18:10:23.438617+0000","last_became_active":"2023-07-14T18:10:22.500219+0000","last_became_peered":"2023-07-14T18:10:22.500219+0000","last_unstale":"2023-07-14T18:10:23.438617+0000","last_undegraded":"2023-07-14T18:10:23.438617+0000","last_fullsized":"2023-07-14T18:10:23.438617+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:09.251191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981056+0000","last_change":"2023-07-14T18:10:17.214307+0000","last_active":"2023-07-14T18:10:36.981056+0000","last_peered":"2023-07-14T18:10:36.981056+0000","last_clean":"2023-07-14T18:10:36.981056+0000","last_became_active":"2023-07-14T18:10:17.214004+0000","last_became_peered":"2023-07-14T18:10:17.214004+0000","last_unstale":"2023-07-14T18:10:36.981056+0000","last_undegraded":"2023-07-14T18:10:36.981056+0000","last_fullsized":"2023-07-14T18:10:36.981056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:50:59.158356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089823+0000","last_change":"2023-07-14T18:10:18.120504+0000","last_active":"2023-07-14T18:10:24.089823+0000","last_peered":"2023-07-14T18:10:24.089823+0000","last_clean":"2023-07-14T18:10:24.089823+0000","last_became_active":"2023-07-14T18:10:18.120133+0000","last_became_peered":"2023-07-14T18:10:18.120133+0000","last_unstale":"2023-07-14T18:10:24.089823+0000","last_undegraded":"2023-07-14T18:10:24.089823+0000","last_fullsized":"2023-07-14T18:10:24.089823+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:54.324914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089420+0000","last_change":"2023-07-14T18:10:22.453040+0000","last_active":"2023-07-14T18:10:24.089420+0000","last_peered":"2023-07-14T18:10:24.089420+0000","last_clean":"2023-07-14T18:10:24.089420+0000","last_became_active":"2023-07-14T18:10:22.452694+0000","last_became_peered":"2023-07-14T18:10:22.452694+0000","last_unstale":"2023-07-14T18:10:24.089420+0000","last_undegraded":"2023-07-14T18:10:24.089420+0000","last_fullsized":"2023-07-14T18:10:24.089420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:43:06.299843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428686+0000","last_change":"2023-07-14T18:10:17.212480+0000","last_active":"2023-07-14T18:10:37.428686+0000","last_peered":"2023-07-14T18:10:37.428686+0000","last_clean":"2023-07-14T18:10:37.428686+0000","last_became_active":"2023-07-14T18:10:17.212080+0000","last_became_peered":"2023-07-14T18:10:17.212080+0000","last_unstale":"2023-07-14T18:10:37.428686+0000","last_undegraded":"2023-07-14T18:10:37.428686+0000","last_fullsized":"2023-07-14T18:10:37.428686+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:19:44.529873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089506+0000","last_change":"2023-07-14T18:10:18.121738+0000","last_active":"2023-07-14T18:10:24.089506+0000","last_peered":"2023-07-14T18:10:24.089506+0000","last_clean":"2023-07-14T18:10:24.089506+0000","last_became_active":"2023-07-14T18:10:18.121186+0000","last_became_peered":"2023-07-14T18:10:18.121186+0000","last_unstale":"2023-07-14T18:10:24.089506+0000","last_undegraded":"2023-07-14T18:10:24.089506+0000","last_fullsized":"2023-07-14T18:10:24.089506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:40:16.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440225+0000","last_change":"2023-07-14T18:10:22.461051+0000","last_active":"2023-07-14T18:10:23.440225+0000","last_peered":"2023-07-14T18:10:23.440225+0000","last_clean":"2023-07-14T18:10:23.440225+0000","last_became_active":"2023-07-14T18:10:22.460663+0000","last_became_peered":"2023-07-14T18:10:22.460663+0000","last_unstale":"2023-07-14T18:10:23.440225+0000","last_undegraded":"2023-07-14T18:10:23.440225+0000","last_fullsized":"2023-07-14T18:10:23.440225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:58:08.355441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090601+0000","last_change":"2023-07-14T18:10:17.211686+0000","last_active":"2023-07-14T18:10:24.090601+0000","last_peered":"2023-07-14T18:10:24.090601+0000","last_clean":"2023-07-14T18:10:24.090601+0000","last_became_active":"2023-07-14T18:10:17.211367+0000","last_became_peered":"2023-07-14T18:10:17.211367+0000","last_unstale":"2023-07-14T18:10:24.090601+0000","last_undegraded":"2023-07-14T18:10:24.090601+0000","last_fullsized":"2023-07-14T18:10:24.090601+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:50.829623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089608+0000","last_change":"2023-07-14T18:10:18.116960+0000","last_active":"2023-07-14T18:10:24.089608+0000","last_peered":"2023-07-14T18:10:24.089608+0000","last_clean":"2023-07-14T18:10:24.089608+0000","last_became_active":"2023-07-14T18:10:18.116416+0000","last_became_peered":"2023-07-14T18:10:18.116416+0000","last_unstale":"2023-07-14T18:10:24.089608+0000","last_undegraded":"2023-07-14T18:10:24.089608+0000","last_fullsized":"2023-07-14T18:10:24.089608+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:33.066056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089552+0000","last_change":"2023-07-14T18:10:22.450045+0000","last_active":"2023-07-14T18:10:24.089552+0000","last_peered":"2023-07-14T18:10:24.089552+0000","last_clean":"2023-07-14T18:10:24.089552+0000","last_became_active":"2023-07-14T18:10:22.449621+0000","last_became_peered":"2023-07-14T18:10:22.449621+0000","last_unstale":"2023-07-14T18:10:24.089552+0000","last_undegraded":"2023-07-14T18:10:24.089552+0000","last_fullsized":"2023-07-14T18:10:24.089552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:48:23.837464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846188+0000","last_change":"2023-07-14T18:10:17.210691+0000","last_active":"2023-07-14T18:10:23.846188+0000","last_peered":"2023-07-14T18:10:23.846188+0000","last_clean":"2023-07-14T18:10:23.846188+0000","last_became_active":"2023-07-14T18:10:17.210391+0000","last_became_peered":"2023-07-14T18:10:17.210391+0000","last_unstale":"2023-07-14T18:10:23.846188+0000","last_undegraded":"2023-07-14T18:10:23.846188+0000","last_fullsized":"2023-07-14T18:10:23.846188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:20:04.361243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089271+0000","last_change":"2023-07-14T18:10:18.115347+0000","last_active":"2023-07-14T18:10:24.089271+0000","last_peered":"2023-07-14T18:10:24.089271+0000","last_clean":"2023-07-14T18:10:24.089271+0000","last_became_active":"2023-07-14T18:10:18.115027+0000","last_became_peered":"2023-07-14T18:10:18.115027+0000","last_unstale":"2023-07-14T18:10:24.089271+0000","last_undegraded":"2023-07-14T18:10:24.089271+0000","last_fullsized":"2023-07-14T18:10:24.089271+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:33:02.993257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088765+0000","last_change":"2023-07-14T18:10:18.103185+0000","last_active":"2023-07-14T18:10:24.088765+0000","last_peered":"2023-07-14T18:10:24.088765+0000","last_clean":"2023-07-14T18:10:24.088765+0000","last_became_active":"2023-07-14T18:10:18.102800+0000","last_became_peered":"2023-07-14T18:10:18.102800+0000","last_unstale":"2023-07-14T18:10:24.088765+0000","last_undegraded":"2023-07-14T18:10:24.088765+0000","last_fullsized":"2023-07-14T18:10:24.088765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:35:22.321627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550057+0000","last_change":"2023-07-14T18:10:18.113501+0000","last_active":"2023-07-14T18:10:23.550057+0000","last_peered":"2023-07-14T18:10:23.550057+0000","last_clean":"2023-07-14T18:10:23.550057+0000","last_became_active":"2023-07-14T18:10:18.113205+0000","last_became_peered":"2023-07-14T18:10:18.113205+0000","last_unstale":"2023-07-14T18:10:23.550057+0000","last_undegraded":"2023-07-14T18:10:23.550057+0000","last_fullsized":"2023-07-14T18:10:23.550057+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:13:56.420803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549722+0000","last_change":"2023-07-14T18:10:18.114282+0000","last_active":"2023-07-14T18:10:23.549722+0000","last_peered":"2023-07-14T18:10:23.549722+0000","last_clean":"2023-07-14T18:10:23.549722+0000","last_became_active":"2023-07-14T18:10:18.113968+0000","last_became_peered":"2023-07-14T18:10:18.113968+0000","last_unstale":"2023-07-14T18:10:23.549722+0000","last_undegraded":"2023-07-14T18:10:23.549722+0000","last_fullsized":"2023-07-14T18:10:23.549722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:36.356073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440436+0000","last_change":"2023-07-14T18:10:18.116781+0000","last_active":"2023-07-14T18:10:23.440436+0000","last_peered":"2023-07-14T18:10:23.440436+0000","last_clean":"2023-07-14T18:10:23.440436+0000","last_became_active":"2023-07-14T18:10:18.116352+0000","last_became_peered":"2023-07-14T18:10:18.116352+0000","last_unstale":"2023-07-14T18:10:23.440436+0000","last_undegraded":"2023-07-14T18:10:23.440436+0000","last_fullsized":"2023-07-14T18:10:23.440436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:48:03.670475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846088+0000","last_change":"2023-07-14T18:10:18.112291+0000","last_active":"2023-07-14T18:10:23.846088+0000","last_peered":"2023-07-14T18:10:23.846088+0000","last_clean":"2023-07-14T18:10:23.846088+0000","last_became_active":"2023-07-14T18:10:18.111989+0000","last_became_peered":"2023-07-14T18:10:18.111989+0000","last_unstale":"2023-07-14T18:10:23.846088+0000","last_undegraded":"2023-07-14T18:10:23.846088+0000","last_fullsized":"2023-07-14T18:10:23.846088+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:11:33.924006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089268+0000","last_change":"2023-07-14T18:10:18.112316+0000","last_active":"2023-07-14T18:10:24.089268+0000","last_peered":"2023-07-14T18:10:24.089268+0000","last_clean":"2023-07-14T18:10:24.089268+0000","last_became_active":"2023-07-14T18:10:18.112030+0000","last_became_peered":"2023-07-14T18:10:18.112030+0000","last_unstale":"2023-07-14T18:10:24.089268+0000","last_undegraded":"2023-07-14T18:10:24.089268+0000","last_fullsized":"2023-07-14T18:10:24.089268+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:14:55.405097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550228+0000","last_change":"2023-07-14T18:10:18.116343+0000","last_active":"2023-07-14T18:10:23.550228+0000","last_peered":"2023-07-14T18:10:23.550228+0000","last_clean":"2023-07-14T18:10:23.550228+0000","last_became_active":"2023-07-14T18:10:18.116041+0000","last_became_peered":"2023-07-14T18:10:18.116041+0000","last_unstale":"2023-07-14T18:10:23.550228+0000","last_undegraded":"2023-07-14T18:10:23.550228+0000","last_fullsized":"2023-07-14T18:10:23.550228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:43:43.627634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089357+0000","last_change":"2023-07-14T18:10:18.117580+0000","last_active":"2023-07-14T18:10:24.089357+0000","last_peered":"2023-07-14T18:10:24.089357+0000","last_clean":"2023-07-14T18:10:24.089357+0000","last_became_active":"2023-07-14T18:10:18.117249+0000","last_became_peered":"2023-07-14T18:10:18.117249+0000","last_unstale":"2023-07-14T18:10:24.089357+0000","last_undegraded":"2023-07-14T18:10:24.089357+0000","last_fullsized":"2023-07-14T18:10:24.089357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:42.562595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089822+0000","last_change":"2023-07-14T18:10:18.107276+0000","last_active":"2023-07-14T18:10:24.089822+0000","last_peered":"2023-07-14T18:10:24.089822+0000","last_clean":"2023-07-14T18:10:24.089822+0000","last_became_active":"2023-07-14T18:10:18.107005+0000","last_became_peered":"2023-07-14T18:10:18.107005+0000","last_unstale":"2023-07-14T18:10:24.089822+0000","last_undegraded":"2023-07-14T18:10:24.089822+0000","last_fullsized":"2023-07-14T18:10:24.089822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:13.352444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089145+0000","last_change":"2023-07-14T18:10:18.107921+0000","last_active":"2023-07-14T18:10:24.089145+0000","last_peered":"2023-07-14T18:10:24.089145+0000","last_clean":"2023-07-14T18:10:24.089145+0000","last_became_active":"2023-07-14T18:10:18.107637+0000","last_became_peered":"2023-07-14T18:10:18.107637+0000","last_unstale":"2023-07-14T18:10:24.089145+0000","last_undegraded":"2023-07-14T18:10:24.089145+0000","last_fullsized":"2023-07-14T18:10:24.089145+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:32.687817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089709+0000","last_change":"2023-07-14T18:10:18.120125+0000","last_active":"2023-07-14T18:10:24.089709+0000","last_peered":"2023-07-14T18:10:24.089709+0000","last_clean":"2023-07-14T18:10:24.089709+0000","last_became_active":"2023-07-14T18:10:18.119814+0000","last_became_peered":"2023-07-14T18:10:18.119814+0000","last_unstale":"2023-07-14T18:10:24.089709+0000","last_undegraded":"2023-07-14T18:10:24.089709+0000","last_fullsized":"2023-07-14T18:10:24.089709+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:37:22.531320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845885+0000","last_change":"2023-07-14T18:10:18.113155+0000","last_active":"2023-07-14T18:10:23.845885+0000","last_peered":"2023-07-14T18:10:23.845885+0000","last_clean":"2023-07-14T18:10:23.845885+0000","last_became_active":"2023-07-14T18:10:18.112837+0000","last_became_peered":"2023-07-14T18:10:18.112837+0000","last_unstale":"2023-07-14T18:10:23.845885+0000","last_undegraded":"2023-07-14T18:10:23.845885+0000","last_fullsized":"2023-07-14T18:10:23.845885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:56:13.124803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088936+0000","last_change":"2023-07-14T18:10:18.112290+0000","last_active":"2023-07-14T18:10:24.088936+0000","last_peered":"2023-07-14T18:10:24.088936+0000","last_clean":"2023-07-14T18:10:24.088936+0000","last_became_active":"2023-07-14T18:10:18.111988+0000","last_became_peered":"2023-07-14T18:10:18.111988+0000","last_unstale":"2023-07-14T18:10:24.088936+0000","last_undegraded":"2023-07-14T18:10:24.088936+0000","last_fullsized":"2023-07-14T18:10:24.088936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:26.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6554,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":96,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":88042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449102,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":10,"up_from":71,"seq":304942678034,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1059825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.67500000000000004}]},{"osd":2,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.69099999999999995}]},{"osd":3,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.74199999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.80700000000000005}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.69299999999999995}]},{"osd":8,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.95799999999999996}]},{"osd":9,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.79500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]}]},{"osd":9,"up_from":66,"seq":283467841559,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":466382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.498}]},{"osd":2,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45300000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.37},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.46100000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.441}]},{"osd":5,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.59799999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.53300000000000003}]},{"osd":10,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.54400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":8,"up_from":58,"seq":249108103195,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":466207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.41499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.80500000000000005}]},{"osd":1,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.72199999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.42999999999999999,"15min":0.42299999999999999},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46899999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.499}]},{"osd":3,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.433,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79700000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.66800000000000004}]},{"osd":4,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64100000000000001,"15min":0.63900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.58499999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.36299999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.53900000000000003}]},{"osd":6,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.52300000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72099999999999997,"15min":0.72599999999999998},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.78600000000000003}]},{"osd":7,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.53100000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.70099999999999996,"15min":0.69299999999999995},"min":{"1min":0.501,"5min":0.439,"15min":0.439},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.85999999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.873}]},{"osd":10,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.91400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364831,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1057864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.41399999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.41399999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.57699999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.432,"15min":0.434},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.623,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58499999999999996,"15min":0.58099999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.84499999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54600000000000004,"15min":0.53800000000000003},"min":{"1min":0.432,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61899999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.50700000000000001,"15min":0.50800000000000001},"min":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.627,"15min":0.622},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82299999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.877},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.67700000000000005},"min":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.90600000000000003,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81799999999999995,"5min":1.024,"15min":1.024},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65000000000000002,"15min":0.64300000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.86499999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.78400000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:10:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.66700000000000004,"15min":0.67300000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.877,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.86099999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.71999999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.93300000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528355,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.311},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.34999999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76300000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.373}]},{"osd":2,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.374,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.442},{"interface":"front","average":{"1min":0.628,"5min":0.62,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.32200000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.42199999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.61399999999999999,"15min":0.60899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.59399999999999997}]},{"osd":5,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.51500000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:10:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.44600000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.94299999999999995,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.51100000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.57499999999999996}]},{"osd":8,"last update":"Fri Jul 14 18:09:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.628},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.495}]},{"osd":9,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54200000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:10:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":504315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.50900000000000001,"15min":0.48299999999999998},"min":{"1min":0.32800000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.51300000000000001,"15min":0.49099999999999999},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.622,"15min":0.60399999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.97799999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.45300000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80600000000000005,"5min":1.337,"15min":1.337},"last":0.71999999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.54600000000000004,"15min":0.51600000000000001},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.43099999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.56799999999999995}]},{"osd":4,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.436,"15min":0.41899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45100000000000001,"15min":0.433},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.65800000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.47199999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92900000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.64300000000000002,"15min":0.62},"min":{"1min":0.45800000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.86599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.86599999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.503,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.48299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89000000000000001,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.79000000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59399999999999997,"15min":0.57999999999999996},"min":{"1min":0.38100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58499999999999996,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.97499999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70899999999999996,"15min":0.69599999999999995},"min":{"1min":0.59399999999999997,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66100000000000003,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.85299999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.92100000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:10:17 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.93600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]}]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":962560,"data_stored":465890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51400000000000001,"15min":0.49099999999999999},"min":{"1min":0.41099999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.51400000000000001,"15min":0.5},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.32000000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:09:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52000000000000002,"15min":0.51100000000000001},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78800000000000003,"5min":0.997,"15min":0.997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.52000000000000002,"15min":0.502},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri Jul 14 18:08:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.39800000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.46400000000000002,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54800000000000004,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.42999999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70999999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:09:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.44700000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.64200000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.34100000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.55500000000000005,"15min":0.54300000000000004},"min":{"1min":0.37,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.60399999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.65700000000000003,"15min":0.65900000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.92100000000000004,"5min":0.998,"15min":0.998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85299999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.51300000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:09:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.69299999999999995}]},{"osd":9,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.63100000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.54600000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":506007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.67500000000000004,"5min":0.995,"15min":0.995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68899999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.54000000000000004}]},{"osd":1,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.39000000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":1.29,"15min":1.29},"last":0.749},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.34899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.01,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76900000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:09:57 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55800000000000005,"15min":0.52800000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.94299999999999995,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.57199999999999995,"15min":0.54800000000000004},"min":{"1min":0.41099999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84899999999999998,"5min":0.996,"15min":0.996},"last":0.63900000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55400000000000005,"15min":0.54400000000000004},"min":{"1min":0.45000000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.69399999999999995}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.498,"15min":0.47499999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.49199999999999999,"15min":0.46400000000000002},"min":{"1min":0.29699999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.82699999999999996,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:09:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.42999999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64100000000000001,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95499999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.871}]},{"osd":7,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.92700000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67400000000000004,"15min":0.66700000000000004},"min":{"1min":0.40699999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":0.89100000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":1.032}]},{"osd":9,"last update":"Fri Jul 14 18:09:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.85499999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.0589999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":3,"up_from":25,"seq":107374182447,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":465573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.33900000000000002,"15min":0.32400000000000001},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.36499999999999999,"15min":0.34499999999999997},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.30199999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54400000000000004,"15min":0.53200000000000003},"min":{"1min":0.375,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.53400000000000003},"min":{"1min":0.42099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.36099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.58999999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.55500000000000005,"15min":0.52300000000000002},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85899999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.39900000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.52800000000000002}]},{"osd":5,"last update":"Fri Jul 14 18:10:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.626,"15min":0.61199999999999999},"min":{"1min":0.47199999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78300000000000003,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.59899999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.52000000000000002,"15min":0.499},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.46800000000000003}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81999999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69799999999999995,"15min":0.68799999999999994},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.51700000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:09:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.64100000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.622}]},{"osd":10,"last update":"Fri Jul 14 18:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.505},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.55100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953516,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":962560,"data_stored":465822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78500000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.378,"15min":0.36099999999999999},"min":{"1min":0.28100000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.39800000000000002,"15min":0.38200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.316}]},{"osd":2,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.58199999999999996,"15min":0.56599999999999995},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92200000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.58199999999999996,"15min":0.56499999999999995},"min":{"1min":0.40899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.54400000000000004},"min":{"1min":0.44700000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78300000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.54000000000000004},"min":{"1min":0.34799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73399999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.49299999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.439,"5min":0.373,"15min":0.373},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64200000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.45000000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88300000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.432}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.47199999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67600000000000005,"15min":0.67300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.87,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.71899999999999997,"15min":0.71299999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.60699999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.58999999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.63}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":5,"up_from":37,"seq":158913789992,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1581056,"data_stored":1057596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.38200000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.78000000000000003,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.39100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78500000000000003,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54500000000000004}]},{"osd":1,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.78200000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.624},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.58499999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.70699999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60299999999999998,"15min":0.60399999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77700000000000002,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.66400000000000003}]},{"osd":4,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.81100000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.90100000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.629,"15min":0.63400000000000001},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.755},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65300000000000002,"15min":0.65400000000000003},"min":{"1min":0.501,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82999999999999996,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.85799999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.47799999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86599999999999999,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45600000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.91800000000000004}]},{"osd":8,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.79900000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:09:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.81499999999999995}]},{"osd":10,"last update":"Fri Jul 14 18:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.871}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":38515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":40207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:10:41.118 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T18:10:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.230+0000 7fd01bcb2700 1 -- 172.21.15.115:0/2787567967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd014100680 msgr2=0x7fd014106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.230+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/2787567967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd014100680 0x7fd014106f40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd00c00b0a0 tx=0x7fd00c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.230+0000 7fd01bcb2700 1 -- 172.21.15.115:0/2787567967 shutdown_connections 2023-07-14T18:10:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.230+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/2787567967 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd014100680 0x7fd014106f40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.230+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/2787567967 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0140feb00 0x7fd0140fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.230+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/2787567967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0140ffcf0 0x7fd014100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.230+0000 7fd01bcb2700 1 -- 172.21.15.115:0/2787567967 >> 172.21.15.115:0/2787567967 conn(0x7fd0140fa100 msgr2=0x7fd0140fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.230+0000 7fd01bcb2700 1 -- 172.21.15.115:0/2787567967 shutdown_connections 2023-07-14T18:10:41.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.230+0000 7fd01bcb2700 1 -- 172.21.15.115:0/2787567967 wait complete. 2023-07-14T18:10:41.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01bcb2700 1 Processor -- start 2023-07-14T18:10:41.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01bcb2700 1 -- start start 2023-07-14T18:10:41.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01bcb2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0140feb00 0x7fd014068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:41.272 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01bcb2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0140ffcf0 0x7fd01419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:41.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01bcb2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd014100680 0x7fd01419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:41.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01bcb2700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01419d450 con 0x7fd0140ffcf0 2023-07-14T18:10:41.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01bcb2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01419d5c0 con 0x7fd014100680 2023-07-14T18:10:41.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01bcb2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01419d780 con 0x7fd0140feb00 2023-07-14T18:10:41.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01924d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0140ffcf0 0x7fd01419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:41.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01924d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0140ffcf0 0x7fd01419a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53122/0 (socket says 172.21.15.115:53122) 2023-07-14T18:10:41.273 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01924d700 1 -- 172.21.15.115:0/1194755744 learned_addr learned my addr 172.21.15.115:0/1194755744 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:41.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01a24f700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd014100680 0x7fd01419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:41.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd019a4e700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0140feb00 0x7fd014068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:41.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01924d700 1 -- 172.21.15.115:0/1194755744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0140feb00 msgr2=0x7fd014068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:41.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01924d700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0140feb00 0x7fd014068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01924d700 1 -- 172.21.15.115:0/1194755744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd014100680 msgr2=0x7fd01419cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:41.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01924d700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd014100680 0x7fd01419cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.231+0000 7fd01924d700 1 -- 172.21.15.115:0/1194755744 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c009d00 con 0x7fd0140ffcf0 2023-07-14T18:10:41.274 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.232+0000 7fd01924d700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0140ffcf0 0x7fd01419a630 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c920 tx=0x7fd01000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:41.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.232+0000 7fd006ffd700 1 -- 172.21.15.115:0/1194755744 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd010007b00 con 0x7fd0140ffcf0 2023-07-14T18:10:41.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.232+0000 7fd006ffd700 1 -- 172.21.15.115:0/1194755744 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd010004cc0 con 0x7fd0140ffcf0 2023-07-14T18:10:41.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.232+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01419da60 con 0x7fd0140ffcf0 2023-07-14T18:10:41.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.232+0000 7fd006ffd700 1 -- 172.21.15.115:0/1194755744 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd010005590 con 0x7fd0140ffcf0 2023-07-14T18:10:41.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.232+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01419df70 con 0x7fd0140ffcf0 2023-07-14T18:10:41.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.233+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd014106050 con 0x7fd0140ffcf0 2023-07-14T18:10:41.275 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.234+0000 7fd006ffd700 1 -- 172.21.15.115:0/1194755744 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fd010004750 con 0x7fd0140ffcf0 2023-07-14T18:10:41.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.234+0000 7fd006ffd700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd0000a11a0 0x7fd0000a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:41.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.234+0000 7fd019a4e700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd0000a11a0 0x7fd0000a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:41.276 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.234+0000 7fd006ffd700 1 -- 172.21.15.115:0/1194755744 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd0100cca70 con 0x7fd0140ffcf0 2023-07-14T18:10:41.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.235+0000 7fd019a4e700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd0000a11a0 0x7fd0000a3650 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd0140ffb50 tx=0x7fd008009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:41.277 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.238+0000 7fd006ffd700 1 -- 172.21.15.115:0/1194755744 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd010095fd0 con 0x7fd0140ffcf0 2023-07-14T18:10:41.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.422+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd014060a80 con 0x7fd0000a11a0 2023-07-14T18:10:41.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.437+0000 7fd006ffd700 1 -- 172.21.15.115:0/1194755744 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418901 (secure 0 0 0) 0x7fd014060a80 con 0x7fd0000a11a0 2023-07-14T18:10:41.480 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:41.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.440+0000 7fd006ffd700 1 -- 172.21.15.115:0/1194755744 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fd010004a00 con 0x7fd0140ffcf0 2023-07-14T18:10:41.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd0000a11a0 msgr2=0x7fd0000a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:41.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd0000a11a0 0x7fd0000a3650 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd0140ffb50 tx=0x7fd008009260 comp rx=0 tx=0).stop 2023-07-14T18:10:41.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0140ffcf0 msgr2=0x7fd01419a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:41.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0140ffcf0 0x7fd01419a630 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c920 tx=0x7fd01000cce0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 shutdown_connections 2023-07-14T18:10:41.484 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd0000a11a0 0x7fd0000a3650 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd014100680 0x7fd01419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0140feb00 0x7fd014068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 --2- 172.21.15.115:0/1194755744 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0140ffcf0 0x7fd01419a630 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:41.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 >> 172.21.15.115:0/1194755744 conn(0x7fd0140fa100 msgr2=0x7fd014103190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:41.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 shutdown_connections 2023-07-14T18:10:41.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:41.443+0000 7fd01bcb2700 1 -- 172.21.15.115:0/1194755744 wait complete. 2023-07-14T18:10:41.485 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-07-14T18:10:41.494 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2023-07-14T18:10:40.885199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596922,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":133,"num_read_kb":116,"num_write":277,"num_write_kb":2971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337420,"kb_used_data":13196,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604404,"statfs":{"total":1151940427776,"available":1151594909696,"internally_reserved":0,"allocated":13512704,"data_stored":7447753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4192,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":51,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003285"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089430+0000","last_change":"2023-07-14T18:10:18.110819+0000","last_active":"2023-07-14T18:10:24.089430+0000","last_peered":"2023-07-14T18:10:24.089430+0000","last_clean":"2023-07-14T18:10:24.089430+0000","last_became_active":"2023-07-14T18:10:18.110180+0000","last_became_peered":"2023-07-14T18:10:18.110180+0000","last_unstale":"2023-07-14T18:10:24.089430+0000","last_undegraded":"2023-07-14T18:10:24.089430+0000","last_fullsized":"2023-07-14T18:10:24.089430+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:08:50.579212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440800+0000","last_change":"2023-07-14T18:10:18.117539+0000","last_active":"2023-07-14T18:10:23.440800+0000","last_peered":"2023-07-14T18:10:23.440800+0000","last_clean":"2023-07-14T18:10:23.440800+0000","last_became_active":"2023-07-14T18:10:18.115956+0000","last_became_peered":"2023-07-14T18:10:18.115956+0000","last_unstale":"2023-07-14T18:10:23.440800+0000","last_undegraded":"2023-07-14T18:10:23.440800+0000","last_fullsized":"2023-07-14T18:10:23.440800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:22.723383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440427+0000","last_change":"2023-07-14T18:10:18.120582+0000","last_active":"2023-07-14T18:10:23.440427+0000","last_peered":"2023-07-14T18:10:23.440427+0000","last_clean":"2023-07-14T18:10:23.440427+0000","last_became_active":"2023-07-14T18:10:18.120296+0000","last_became_peered":"2023-07-14T18:10:18.120296+0000","last_unstale":"2023-07-14T18:10:23.440427+0000","last_undegraded":"2023-07-14T18:10:23.440427+0000","last_fullsized":"2023-07-14T18:10:23.440427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:54.381627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438612+0000","last_change":"2023-07-14T18:10:18.112250+0000","last_active":"2023-07-14T18:10:23.438612+0000","last_peered":"2023-07-14T18:10:23.438612+0000","last_clean":"2023-07-14T18:10:23.438612+0000","last_became_active":"2023-07-14T18:10:18.111514+0000","last_became_peered":"2023-07-14T18:10:18.111514+0000","last_unstale":"2023-07-14T18:10:23.438612+0000","last_undegraded":"2023-07-14T18:10:23.438612+0000","last_fullsized":"2023-07-14T18:10:23.438612+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:02:30.280559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438548+0000","last_change":"2023-07-14T18:10:18.111127+0000","last_active":"2023-07-14T18:10:23.438548+0000","last_peered":"2023-07-14T18:10:23.438548+0000","last_clean":"2023-07-14T18:10:23.438548+0000","last_became_active":"2023-07-14T18:10:18.110816+0000","last_became_peered":"2023-07-14T18:10:18.110816+0000","last_unstale":"2023-07-14T18:10:23.438548+0000","last_undegraded":"2023-07-14T18:10:23.438548+0000","last_fullsized":"2023-07-14T18:10:23.438548+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:05:59.027134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734705+0000","last_change":"2023-07-14T18:10:18.113436+0000","last_active":"2023-07-14T18:10:23.734705+0000","last_peered":"2023-07-14T18:10:23.734705+0000","last_clean":"2023-07-14T18:10:23.734705+0000","last_became_active":"2023-07-14T18:10:18.113172+0000","last_became_peered":"2023-07-14T18:10:18.113172+0000","last_unstale":"2023-07-14T18:10:23.734705+0000","last_undegraded":"2023-07-14T18:10:23.734705+0000","last_fullsized":"2023-07-14T18:10:23.734705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:33:57.446784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089034+0000","last_change":"2023-07-14T18:10:18.116720+0000","last_active":"2023-07-14T18:10:24.089034+0000","last_peered":"2023-07-14T18:10:24.089034+0000","last_clean":"2023-07-14T18:10:24.089034+0000","last_became_active":"2023-07-14T18:10:18.116361+0000","last_became_peered":"2023-07-14T18:10:18.116361+0000","last_unstale":"2023-07-14T18:10:24.089034+0000","last_undegraded":"2023-07-14T18:10:24.089034+0000","last_fullsized":"2023-07-14T18:10:24.089034+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:54.195458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088667+0000","last_change":"2023-07-14T18:10:18.111981+0000","last_active":"2023-07-14T18:10:24.088667+0000","last_peered":"2023-07-14T18:10:24.088667+0000","last_clean":"2023-07-14T18:10:24.088667+0000","last_became_active":"2023-07-14T18:10:18.111666+0000","last_became_peered":"2023-07-14T18:10:18.111666+0000","last_unstale":"2023-07-14T18:10:24.088667+0000","last_undegraded":"2023-07-14T18:10:24.088667+0000","last_fullsized":"2023-07-14T18:10:24.088667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:07:35.710429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089557+0000","last_change":"2023-07-14T18:10:18.119497+0000","last_active":"2023-07-14T18:10:24.089557+0000","last_peered":"2023-07-14T18:10:24.089557+0000","last_clean":"2023-07-14T18:10:24.089557+0000","last_became_active":"2023-07-14T18:10:18.119090+0000","last_became_peered":"2023-07-14T18:10:18.119090+0000","last_unstale":"2023-07-14T18:10:24.089557+0000","last_undegraded":"2023-07-14T18:10:24.089557+0000","last_fullsized":"2023-07-14T18:10:24.089557+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:15:13.934598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.735011+0000","last_change":"2023-07-14T18:10:18.107353+0000","last_active":"2023-07-14T18:10:23.735011+0000","last_peered":"2023-07-14T18:10:23.735011+0000","last_clean":"2023-07-14T18:10:23.735011+0000","last_became_active":"2023-07-14T18:10:18.106933+0000","last_became_peered":"2023-07-14T18:10:18.106933+0000","last_unstale":"2023-07-14T18:10:23.735011+0000","last_undegraded":"2023-07-14T18:10:23.735011+0000","last_fullsized":"2023-07-14T18:10:23.735011+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:20:25.508991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089317+0000","last_change":"2023-07-14T18:10:18.121064+0000","last_active":"2023-07-14T18:10:24.089317+0000","last_peered":"2023-07-14T18:10:24.089317+0000","last_clean":"2023-07-14T18:10:24.089317+0000","last_became_active":"2023-07-14T18:10:18.120758+0000","last_became_peered":"2023-07-14T18:10:18.120758+0000","last_unstale":"2023-07-14T18:10:24.089317+0000","last_undegraded":"2023-07-14T18:10:24.089317+0000","last_fullsized":"2023-07-14T18:10:24.089317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:42.972896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438568+0000","last_change":"2023-07-14T18:10:18.112436+0000","last_active":"2023-07-14T18:10:23.438568+0000","last_peered":"2023-07-14T18:10:23.438568+0000","last_clean":"2023-07-14T18:10:23.438568+0000","last_became_active":"2023-07-14T18:10:18.111942+0000","last_became_peered":"2023-07-14T18:10:18.111942+0000","last_unstale":"2023-07-14T18:10:23.438568+0000","last_undegraded":"2023-07-14T18:10:23.438568+0000","last_fullsized":"2023-07-14T18:10:23.438568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:57.856511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845884+0000","last_change":"2023-07-14T18:10:18.106134+0000","last_active":"2023-07-14T18:10:23.845884+0000","last_peered":"2023-07-14T18:10:23.845884+0000","last_clean":"2023-07-14T18:10:23.845884+0000","last_became_active":"2023-07-14T18:10:18.105481+0000","last_became_peered":"2023-07-14T18:10:18.105481+0000","last_unstale":"2023-07-14T18:10:23.845884+0000","last_undegraded":"2023-07-14T18:10:23.845884+0000","last_fullsized":"2023-07-14T18:10:23.845884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:53.958762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440197+0000","last_change":"2023-07-14T18:10:18.117399+0000","last_active":"2023-07-14T18:10:23.440197+0000","last_peered":"2023-07-14T18:10:23.440197+0000","last_clean":"2023-07-14T18:10:23.440197+0000","last_became_active":"2023-07-14T18:10:18.117094+0000","last_became_peered":"2023-07-14T18:10:18.117094+0000","last_unstale":"2023-07-14T18:10:23.440197+0000","last_undegraded":"2023-07-14T18:10:23.440197+0000","last_fullsized":"2023-07-14T18:10:23.440197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:10:09.445305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440157+0000","last_change":"2023-07-14T18:10:18.117150+0000","last_active":"2023-07-14T18:10:23.440157+0000","last_peered":"2023-07-14T18:10:23.440157+0000","last_clean":"2023-07-14T18:10:23.440157+0000","last_became_active":"2023-07-14T18:10:18.116805+0000","last_became_peered":"2023-07-14T18:10:18.116805+0000","last_unstale":"2023-07-14T18:10:23.440157+0000","last_undegraded":"2023-07-14T18:10:23.440157+0000","last_fullsized":"2023-07-14T18:10:23.440157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:48:43.803465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088549+0000","last_change":"2023-07-14T18:10:18.120868+0000","last_active":"2023-07-14T18:10:24.088549+0000","last_peered":"2023-07-14T18:10:24.088549+0000","last_clean":"2023-07-14T18:10:24.088549+0000","last_became_active":"2023-07-14T18:10:18.120083+0000","last_became_peered":"2023-07-14T18:10:18.120083+0000","last_unstale":"2023-07-14T18:10:24.088549+0000","last_undegraded":"2023-07-14T18:10:24.088549+0000","last_fullsized":"2023-07-14T18:10:24.088549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:50:07.162775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089298+0000","last_change":"2023-07-14T18:10:18.111011+0000","last_active":"2023-07-14T18:10:24.089298+0000","last_peered":"2023-07-14T18:10:24.089298+0000","last_clean":"2023-07-14T18:10:24.089298+0000","last_became_active":"2023-07-14T18:10:18.110517+0000","last_became_peered":"2023-07-14T18:10:18.110517+0000","last_unstale":"2023-07-14T18:10:24.089298+0000","last_undegraded":"2023-07-14T18:10:24.089298+0000","last_fullsized":"2023-07-14T18:10:24.089298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:00:45.278711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089798+0000","last_change":"2023-07-14T18:10:18.110739+0000","last_active":"2023-07-14T18:10:24.089798+0000","last_peered":"2023-07-14T18:10:24.089798+0000","last_clean":"2023-07-14T18:10:24.089798+0000","last_became_active":"2023-07-14T18:10:18.110442+0000","last_became_peered":"2023-07-14T18:10:18.110442+0000","last_unstale":"2023-07-14T18:10:24.089798+0000","last_undegraded":"2023-07-14T18:10:24.089798+0000","last_fullsized":"2023-07-14T18:10:24.089798+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:45.130540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550066+0000","last_change":"2023-07-14T18:10:18.109955+0000","last_active":"2023-07-14T18:10:23.550066+0000","last_peered":"2023-07-14T18:10:23.550066+0000","last_clean":"2023-07-14T18:10:23.550066+0000","last_became_active":"2023-07-14T18:10:18.109651+0000","last_became_peered":"2023-07-14T18:10:18.109651+0000","last_unstale":"2023-07-14T18:10:23.550066+0000","last_undegraded":"2023-07-14T18:10:23.550066+0000","last_fullsized":"2023-07-14T18:10:23.550066+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:22:46.479116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549977+0000","last_change":"2023-07-14T18:10:22.457996+0000","last_active":"2023-07-14T18:10:23.549977+0000","last_peered":"2023-07-14T18:10:23.549977+0000","last_clean":"2023-07-14T18:10:23.549977+0000","last_became_active":"2023-07-14T18:10:22.457573+0000","last_became_peered":"2023-07-14T18:10:22.457573+0000","last_unstale":"2023-07-14T18:10:23.549977+0000","last_undegraded":"2023-07-14T18:10:23.549977+0000","last_fullsized":"2023-07-14T18:10:23.549977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:17.148116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090026+0000","last_change":"2023-07-14T18:10:17.216780+0000","last_active":"2023-07-14T18:10:24.090026+0000","last_peered":"2023-07-14T18:10:24.090026+0000","last_clean":"2023-07-14T18:10:24.090026+0000","last_became_active":"2023-07-14T18:10:17.216423+0000","last_became_peered":"2023-07-14T18:10:17.216423+0000","last_unstale":"2023-07-14T18:10:24.090026+0000","last_undegraded":"2023-07-14T18:10:24.090026+0000","last_fullsized":"2023-07-14T18:10:24.090026+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:24.522131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089127+0000","last_change":"2023-07-14T18:10:18.108615+0000","last_active":"2023-07-14T18:10:24.089127+0000","last_peered":"2023-07-14T18:10:24.089127+0000","last_clean":"2023-07-14T18:10:24.089127+0000","last_became_active":"2023-07-14T18:10:18.108316+0000","last_became_peered":"2023-07-14T18:10:18.108316+0000","last_unstale":"2023-07-14T18:10:24.089127+0000","last_undegraded":"2023-07-14T18:10:24.089127+0000","last_fullsized":"2023-07-14T18:10:24.089127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:27:25.084342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438203+0000","last_change":"2023-07-14T18:10:22.453040+0000","last_active":"2023-07-14T18:10:23.438203+0000","last_peered":"2023-07-14T18:10:23.438203+0000","last_clean":"2023-07-14T18:10:23.438203+0000","last_became_active":"2023-07-14T18:10:22.452643+0000","last_became_peered":"2023-07-14T18:10:22.452643+0000","last_unstale":"2023-07-14T18:10:23.438203+0000","last_undegraded":"2023-07-14T18:10:23.438203+0000","last_fullsized":"2023-07-14T18:10:23.438203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:18.191433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089404+0000","last_change":"2023-07-14T18:10:17.214085+0000","last_active":"2023-07-14T18:10:24.089404+0000","last_peered":"2023-07-14T18:10:24.089404+0000","last_clean":"2023-07-14T18:10:24.089404+0000","last_became_active":"2023-07-14T18:10:17.213772+0000","last_became_peered":"2023-07-14T18:10:17.213772+0000","last_unstale":"2023-07-14T18:10:24.089404+0000","last_undegraded":"2023-07-14T18:10:24.089404+0000","last_fullsized":"2023-07-14T18:10:24.089404+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:16:35.586284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440705+0000","last_change":"2023-07-14T18:10:18.120532+0000","last_active":"2023-07-14T18:10:23.440705+0000","last_peered":"2023-07-14T18:10:23.440705+0000","last_clean":"2023-07-14T18:10:23.440705+0000","last_became_active":"2023-07-14T18:10:18.120199+0000","last_became_peered":"2023-07-14T18:10:18.120199+0000","last_unstale":"2023-07-14T18:10:23.440705+0000","last_undegraded":"2023-07-14T18:10:23.440705+0000","last_fullsized":"2023-07-14T18:10:23.440705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:05:38.292570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438439+0000","last_change":"2023-07-14T18:10:22.458820+0000","last_active":"2023-07-14T18:10:23.438439+0000","last_peered":"2023-07-14T18:10:23.438439+0000","last_clean":"2023-07-14T18:10:23.438439+0000","last_became_active":"2023-07-14T18:10:22.458480+0000","last_became_peered":"2023-07-14T18:10:22.458480+0000","last_unstale":"2023-07-14T18:10:23.438439+0000","last_undegraded":"2023-07-14T18:10:23.438439+0000","last_fullsized":"2023-07-14T18:10:23.438439+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:45.901536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428015+0000","last_change":"2023-07-14T18:10:17.212329+0000","last_active":"2023-07-14T18:10:37.428015+0000","last_peered":"2023-07-14T18:10:37.428015+0000","last_clean":"2023-07-14T18:10:37.428015+0000","last_became_active":"2023-07-14T18:10:17.212013+0000","last_became_peered":"2023-07-14T18:10:17.212013+0000","last_unstale":"2023-07-14T18:10:37.428015+0000","last_undegraded":"2023-07-14T18:10:37.428015+0000","last_fullsized":"2023-07-14T18:10:37.428015+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:19:15.950647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089229+0000","last_change":"2023-07-14T18:10:18.114621+0000","last_active":"2023-07-14T18:10:24.089229+0000","last_peered":"2023-07-14T18:10:24.089229+0000","last_clean":"2023-07-14T18:10:24.089229+0000","last_became_active":"2023-07-14T18:10:18.114330+0000","last_became_peered":"2023-07-14T18:10:18.114330+0000","last_unstale":"2023-07-14T18:10:24.089229+0000","last_undegraded":"2023-07-14T18:10:24.089229+0000","last_fullsized":"2023-07-14T18:10:24.089229+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:47:21.060539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438244+0000","last_change":"2023-07-14T18:10:22.453179+0000","last_active":"2023-07-14T18:10:23.438244+0000","last_peered":"2023-07-14T18:10:23.438244+0000","last_clean":"2023-07-14T18:10:23.438244+0000","last_became_active":"2023-07-14T18:10:22.452745+0000","last_became_peered":"2023-07-14T18:10:22.452745+0000","last_unstale":"2023-07-14T18:10:23.438244+0000","last_undegraded":"2023-07-14T18:10:23.438244+0000","last_fullsized":"2023-07-14T18:10:23.438244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:22:16.784956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438868+0000","last_change":"2023-07-14T18:10:17.215115+0000","last_active":"2023-07-14T18:10:23.438868+0000","last_peered":"2023-07-14T18:10:23.438868+0000","last_clean":"2023-07-14T18:10:23.438868+0000","last_became_active":"2023-07-14T18:10:17.214813+0000","last_became_peered":"2023-07-14T18:10:17.214813+0000","last_unstale":"2023-07-14T18:10:23.438868+0000","last_undegraded":"2023-07-14T18:10:23.438868+0000","last_fullsized":"2023-07-14T18:10:23.438868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:48:32.267436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090280+0000","last_change":"2023-07-14T18:10:18.119987+0000","last_active":"2023-07-14T18:10:24.090280+0000","last_peered":"2023-07-14T18:10:24.090280+0000","last_clean":"2023-07-14T18:10:24.090280+0000","last_became_active":"2023-07-14T18:10:18.119695+0000","last_became_peered":"2023-07-14T18:10:18.119695+0000","last_unstale":"2023-07-14T18:10:24.090280+0000","last_undegraded":"2023-07-14T18:10:24.090280+0000","last_fullsized":"2023-07-14T18:10:24.090280+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:46.295719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550007+0000","last_change":"2023-07-14T18:10:22.449689+0000","last_active":"2023-07-14T18:10:23.550007+0000","last_peered":"2023-07-14T18:10:23.550007+0000","last_clean":"2023-07-14T18:10:23.550007+0000","last_became_active":"2023-07-14T18:10:22.449273+0000","last_became_peered":"2023-07-14T18:10:22.449273+0000","last_unstale":"2023-07-14T18:10:23.550007+0000","last_undegraded":"2023-07-14T18:10:23.550007+0000","last_fullsized":"2023-07-14T18:10:23.550007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:54:23.642287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.982902+0000","last_change":"2023-07-14T18:10:17.216438+0000","last_active":"2023-07-14T18:10:36.982902+0000","last_peered":"2023-07-14T18:10:36.982902+0000","last_clean":"2023-07-14T18:10:36.982902+0000","last_became_active":"2023-07-14T18:10:17.216169+0000","last_became_peered":"2023-07-14T18:10:17.216169+0000","last_unstale":"2023-07-14T18:10:36.982902+0000","last_undegraded":"2023-07-14T18:10:36.982902+0000","last_fullsized":"2023-07-14T18:10:36.982902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:39.977001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438762+0000","last_change":"2023-07-14T18:10:18.112917+0000","last_active":"2023-07-14T18:10:23.438762+0000","last_peered":"2023-07-14T18:10:23.438762+0000","last_clean":"2023-07-14T18:10:23.438762+0000","last_became_active":"2023-07-14T18:10:18.112631+0000","last_became_peered":"2023-07-14T18:10:18.112631+0000","last_unstale":"2023-07-14T18:10:23.438762+0000","last_undegraded":"2023-07-14T18:10:23.438762+0000","last_fullsized":"2023-07-14T18:10:23.438762+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:14:01.774309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549936+0000","last_change":"2023-07-14T18:10:22.454278+0000","last_active":"2023-07-14T18:10:23.549936+0000","last_peered":"2023-07-14T18:10:23.549936+0000","last_clean":"2023-07-14T18:10:23.549936+0000","last_became_active":"2023-07-14T18:10:22.453834+0000","last_became_peered":"2023-07-14T18:10:22.453834+0000","last_unstale":"2023-07-14T18:10:23.549936+0000","last_undegraded":"2023-07-14T18:10:23.549936+0000","last_fullsized":"2023-07-14T18:10:23.549936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:31:24.983615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734643+0000","last_change":"2023-07-14T18:10:17.210217+0000","last_active":"2023-07-14T18:10:23.734643+0000","last_peered":"2023-07-14T18:10:23.734643+0000","last_clean":"2023-07-14T18:10:23.734643+0000","last_became_active":"2023-07-14T18:10:17.209953+0000","last_became_peered":"2023-07-14T18:10:17.209953+0000","last_unstale":"2023-07-14T18:10:23.734643+0000","last_undegraded":"2023-07-14T18:10:23.734643+0000","last_fullsized":"2023-07-14T18:10:23.734643+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:19:46.091265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090402+0000","last_change":"2023-07-14T18:10:18.113009+0000","last_active":"2023-07-14T18:10:24.090402+0000","last_peered":"2023-07-14T18:10:24.090402+0000","last_clean":"2023-07-14T18:10:24.090402+0000","last_became_active":"2023-07-14T18:10:18.112695+0000","last_became_peered":"2023-07-14T18:10:18.112695+0000","last_unstale":"2023-07-14T18:10:24.090402+0000","last_undegraded":"2023-07-14T18:10:24.090402+0000","last_fullsized":"2023-07-14T18:10:24.090402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:06:04.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089159+0000","last_change":"2023-07-14T18:10:22.457687+0000","last_active":"2023-07-14T18:10:24.089159+0000","last_peered":"2023-07-14T18:10:24.089159+0000","last_clean":"2023-07-14T18:10:24.089159+0000","last_became_active":"2023-07-14T18:10:22.457270+0000","last_became_peered":"2023-07-14T18:10:22.457270+0000","last_unstale":"2023-07-14T18:10:24.089159+0000","last_undegraded":"2023-07-14T18:10:24.089159+0000","last_fullsized":"2023-07-14T18:10:24.089159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:37:58.192527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429469+0000","last_change":"2023-07-14T18:10:17.209638+0000","last_active":"2023-07-14T18:10:37.429469+0000","last_peered":"2023-07-14T18:10:37.429469+0000","last_clean":"2023-07-14T18:10:37.429469+0000","last_became_active":"2023-07-14T18:10:17.209409+0000","last_became_peered":"2023-07-14T18:10:17.209409+0000","last_unstale":"2023-07-14T18:10:37.429469+0000","last_undegraded":"2023-07-14T18:10:37.429469+0000","last_fullsized":"2023-07-14T18:10:37.429469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:32.419102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088533+0000","last_change":"2023-07-14T18:10:18.109479+0000","last_active":"2023-07-14T18:10:24.088533+0000","last_peered":"2023-07-14T18:10:24.088533+0000","last_clean":"2023-07-14T18:10:24.088533+0000","last_became_active":"2023-07-14T18:10:18.109161+0000","last_became_peered":"2023-07-14T18:10:18.109161+0000","last_unstale":"2023-07-14T18:10:24.088533+0000","last_undegraded":"2023-07-14T18:10:24.088533+0000","last_fullsized":"2023-07-14T18:10:24.088533+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:05.204295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440596+0000","last_change":"2023-07-14T18:10:22.453731+0000","last_active":"2023-07-14T18:10:23.440596+0000","last_peered":"2023-07-14T18:10:23.440596+0000","last_clean":"2023-07-14T18:10:23.440596+0000","last_became_active":"2023-07-14T18:10:22.453311+0000","last_became_peered":"2023-07-14T18:10:22.453311+0000","last_unstale":"2023-07-14T18:10:23.440596+0000","last_undegraded":"2023-07-14T18:10:23.440596+0000","last_fullsized":"2023-07-14T18:10:23.440596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:35.892698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089898+0000","last_change":"2023-07-14T18:10:17.218050+0000","last_active":"2023-07-14T18:10:24.089898+0000","last_peered":"2023-07-14T18:10:24.089898+0000","last_clean":"2023-07-14T18:10:24.089898+0000","last_became_active":"2023-07-14T18:10:17.217793+0000","last_became_peered":"2023-07-14T18:10:17.217793+0000","last_unstale":"2023-07-14T18:10:24.089898+0000","last_undegraded":"2023-07-14T18:10:24.089898+0000","last_fullsized":"2023-07-14T18:10:24.089898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:23.781098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089467+0000","last_change":"2023-07-14T18:10:18.111598+0000","last_active":"2023-07-14T18:10:24.089467+0000","last_peered":"2023-07-14T18:10:24.089467+0000","last_clean":"2023-07-14T18:10:24.089467+0000","last_became_active":"2023-07-14T18:10:18.111244+0000","last_became_peered":"2023-07-14T18:10:18.111244+0000","last_unstale":"2023-07-14T18:10:24.089467+0000","last_undegraded":"2023-07-14T18:10:24.089467+0000","last_fullsized":"2023-07-14T18:10:24.089467+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:26:47.187587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088793+0000","last_change":"2023-07-14T18:10:22.447676+0000","last_active":"2023-07-14T18:10:24.088793+0000","last_peered":"2023-07-14T18:10:24.088793+0000","last_clean":"2023-07-14T18:10:24.088793+0000","last_became_active":"2023-07-14T18:10:22.446648+0000","last_became_peered":"2023-07-14T18:10:22.446648+0000","last_unstale":"2023-07-14T18:10:24.088793+0000","last_undegraded":"2023-07-14T18:10:24.088793+0000","last_fullsized":"2023-07-14T18:10:24.088793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:17:30.941899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.735022+0000","last_change":"2023-07-14T18:10:17.213897+0000","last_active":"2023-07-14T18:10:23.735022+0000","last_peered":"2023-07-14T18:10:23.735022+0000","last_clean":"2023-07-14T18:10:23.735022+0000","last_became_active":"2023-07-14T18:10:17.213547+0000","last_became_peered":"2023-07-14T18:10:17.213547+0000","last_unstale":"2023-07-14T18:10:23.735022+0000","last_undegraded":"2023-07-14T18:10:23.735022+0000","last_fullsized":"2023-07-14T18:10:23.735022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:44.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089327+0000","last_change":"2023-07-14T18:10:18.114776+0000","last_active":"2023-07-14T18:10:24.089327+0000","last_peered":"2023-07-14T18:10:24.089327+0000","last_clean":"2023-07-14T18:10:24.089327+0000","last_became_active":"2023-07-14T18:10:18.112469+0000","last_became_peered":"2023-07-14T18:10:18.112469+0000","last_unstale":"2023-07-14T18:10:24.089327+0000","last_undegraded":"2023-07-14T18:10:24.089327+0000","last_fullsized":"2023-07-14T18:10:24.089327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:30.864496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088873+0000","last_change":"2023-07-14T18:10:22.454215+0000","last_active":"2023-07-14T18:10:24.088873+0000","last_peered":"2023-07-14T18:10:24.088873+0000","last_clean":"2023-07-14T18:10:24.088873+0000","last_became_active":"2023-07-14T18:10:22.453889+0000","last_became_peered":"2023-07-14T18:10:22.453889+0000","last_unstale":"2023-07-14T18:10:24.088873+0000","last_undegraded":"2023-07-14T18:10:24.088873+0000","last_fullsized":"2023-07-14T18:10:24.088873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:30:09.159482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981368+0000","last_change":"2023-07-14T18:10:17.212602+0000","last_active":"2023-07-14T18:10:36.981368+0000","last_peered":"2023-07-14T18:10:36.981368+0000","last_clean":"2023-07-14T18:10:36.981368+0000","last_became_active":"2023-07-14T18:10:17.212087+0000","last_became_peered":"2023-07-14T18:10:17.212087+0000","last_unstale":"2023-07-14T18:10:36.981368+0000","last_undegraded":"2023-07-14T18:10:36.981368+0000","last_fullsized":"2023-07-14T18:10:36.981368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:30:00.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440688+0000","last_change":"2023-07-14T18:10:18.116972+0000","last_active":"2023-07-14T18:10:23.440688+0000","last_peered":"2023-07-14T18:10:23.440688+0000","last_clean":"2023-07-14T18:10:23.440688+0000","last_became_active":"2023-07-14T18:10:18.116423+0000","last_became_peered":"2023-07-14T18:10:18.116423+0000","last_unstale":"2023-07-14T18:10:23.440688+0000","last_undegraded":"2023-07-14T18:10:23.440688+0000","last_fullsized":"2023-07-14T18:10:23.440688+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:53.315535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438563+0000","last_change":"2023-07-14T18:10:22.450207+0000","last_active":"2023-07-14T18:10:23.438563+0000","last_peered":"2023-07-14T18:10:23.438563+0000","last_clean":"2023-07-14T18:10:23.438563+0000","last_became_active":"2023-07-14T18:10:22.449826+0000","last_became_peered":"2023-07-14T18:10:22.449826+0000","last_unstale":"2023-07-14T18:10:23.438563+0000","last_undegraded":"2023-07-14T18:10:23.438563+0000","last_fullsized":"2023-07-14T18:10:23.438563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:18.013945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090602+0000","last_change":"2023-07-14T18:10:17.213769+0000","last_active":"2023-07-14T18:10:24.090602+0000","last_peered":"2023-07-14T18:10:24.090602+0000","last_clean":"2023-07-14T18:10:24.090602+0000","last_became_active":"2023-07-14T18:10:17.213465+0000","last_became_peered":"2023-07-14T18:10:17.213465+0000","last_unstale":"2023-07-14T18:10:24.090602+0000","last_undegraded":"2023-07-14T18:10:24.090602+0000","last_fullsized":"2023-07-14T18:10:24.090602+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:38:07.890716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089153+0000","last_change":"2023-07-14T18:10:18.122764+0000","last_active":"2023-07-14T18:10:24.089153+0000","last_peered":"2023-07-14T18:10:24.089153+0000","last_clean":"2023-07-14T18:10:24.089153+0000","last_became_active":"2023-07-14T18:10:18.120522+0000","last_became_peered":"2023-07-14T18:10:18.120522+0000","last_unstale":"2023-07-14T18:10:24.089153+0000","last_undegraded":"2023-07-14T18:10:24.089153+0000","last_fullsized":"2023-07-14T18:10:24.089153+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:54:07.567008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089725+0000","last_change":"2023-07-14T18:10:22.455635+0000","last_active":"2023-07-14T18:10:24.089725+0000","last_peered":"2023-07-14T18:10:24.089725+0000","last_clean":"2023-07-14T18:10:24.089725+0000","last_became_active":"2023-07-14T18:10:22.455128+0000","last_became_peered":"2023-07-14T18:10:22.455128+0000","last_unstale":"2023-07-14T18:10:24.089725+0000","last_undegraded":"2023-07-14T18:10:24.089725+0000","last_fullsized":"2023-07-14T18:10:24.089725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:01.364723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'10","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:34.473808+0000","last_change":"2023-07-14T18:10:17.215255+0000","last_active":"2023-07-14T18:10:34.473808+0000","last_peered":"2023-07-14T18:10:34.473808+0000","last_clean":"2023-07-14T18:10:34.473808+0000","last_became_active":"2023-07-14T18:10:17.214900+0000","last_became_peered":"2023-07-14T18:10:17.214900+0000","last_unstale":"2023-07-14T18:10:34.473808+0000","last_undegraded":"2023-07-14T18:10:34.473808+0000","last_fullsized":"2023-07-14T18:10:34.473808+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:01.480644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438902+0000","last_change":"2023-07-14T18:10:18.115355+0000","last_active":"2023-07-14T18:10:23.438902+0000","last_peered":"2023-07-14T18:10:23.438902+0000","last_clean":"2023-07-14T18:10:23.438902+0000","last_became_active":"2023-07-14T18:10:18.115023+0000","last_became_peered":"2023-07-14T18:10:18.115023+0000","last_unstale":"2023-07-14T18:10:23.438902+0000","last_undegraded":"2023-07-14T18:10:23.438902+0000","last_fullsized":"2023-07-14T18:10:23.438902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:16.057123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090803+0000","last_change":"2023-07-14T18:10:22.453113+0000","last_active":"2023-07-14T18:10:24.090803+0000","last_peered":"2023-07-14T18:10:24.090803+0000","last_clean":"2023-07-14T18:10:24.090803+0000","last_became_active":"2023-07-14T18:10:22.452688+0000","last_became_peered":"2023-07-14T18:10:22.452688+0000","last_unstale":"2023-07-14T18:10:24.090803+0000","last_undegraded":"2023-07-14T18:10:24.090803+0000","last_fullsized":"2023-07-14T18:10:24.090803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:47.829022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430020+0000","last_change":"2023-07-14T18:10:17.210924+0000","last_active":"2023-07-14T18:10:37.430020+0000","last_peered":"2023-07-14T18:10:37.430020+0000","last_clean":"2023-07-14T18:10:37.430020+0000","last_became_active":"2023-07-14T18:10:17.210604+0000","last_became_peered":"2023-07-14T18:10:17.210604+0000","last_unstale":"2023-07-14T18:10:37.430020+0000","last_undegraded":"2023-07-14T18:10:37.430020+0000","last_fullsized":"2023-07-14T18:10:37.430020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:10:28.635558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845862+0000","last_change":"2023-07-14T18:10:18.115094+0000","last_active":"2023-07-14T18:10:23.845862+0000","last_peered":"2023-07-14T18:10:23.845862+0000","last_clean":"2023-07-14T18:10:23.845862+0000","last_became_active":"2023-07-14T18:10:18.114538+0000","last_became_peered":"2023-07-14T18:10:18.114538+0000","last_unstale":"2023-07-14T18:10:23.845862+0000","last_undegraded":"2023-07-14T18:10:23.845862+0000","last_fullsized":"2023-07-14T18:10:23.845862+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:19:15.916223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088869+0000","last_change":"2023-07-14T18:10:22.459151+0000","last_active":"2023-07-14T18:10:24.088869+0000","last_peered":"2023-07-14T18:10:24.088869+0000","last_clean":"2023-07-14T18:10:24.088869+0000","last_became_active":"2023-07-14T18:10:22.458550+0000","last_became_peered":"2023-07-14T18:10:22.458550+0000","last_unstale":"2023-07-14T18:10:24.088869+0000","last_undegraded":"2023-07-14T18:10:24.088869+0000","last_fullsized":"2023-07-14T18:10:24.088869+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:11.750342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.982150+0000","last_change":"2023-07-14T18:10:17.216475+0000","last_active":"2023-07-14T18:10:36.982150+0000","last_peered":"2023-07-14T18:10:36.982150+0000","last_clean":"2023-07-14T18:10:36.982150+0000","last_became_active":"2023-07-14T18:10:17.216148+0000","last_became_peered":"2023-07-14T18:10:17.216148+0000","last_unstale":"2023-07-14T18:10:36.982150+0000","last_undegraded":"2023-07-14T18:10:36.982150+0000","last_fullsized":"2023-07-14T18:10:36.982150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:00:50.106730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089545+0000","last_change":"2023-07-14T18:10:18.121764+0000","last_active":"2023-07-14T18:10:24.089545+0000","last_peered":"2023-07-14T18:10:24.089545+0000","last_clean":"2023-07-14T18:10:24.089545+0000","last_became_active":"2023-07-14T18:10:18.121458+0000","last_became_peered":"2023-07-14T18:10:18.121458+0000","last_unstale":"2023-07-14T18:10:24.089545+0000","last_undegraded":"2023-07-14T18:10:24.089545+0000","last_fullsized":"2023-07-14T18:10:24.089545+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:26:22.145994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089867+0000","last_change":"2023-07-14T18:10:22.457737+0000","last_active":"2023-07-14T18:10:24.089867+0000","last_peered":"2023-07-14T18:10:24.089867+0000","last_clean":"2023-07-14T18:10:24.089867+0000","last_became_active":"2023-07-14T18:10:22.457359+0000","last_became_peered":"2023-07-14T18:10:22.457359+0000","last_unstale":"2023-07-14T18:10:24.089867+0000","last_undegraded":"2023-07-14T18:10:24.089867+0000","last_fullsized":"2023-07-14T18:10:24.089867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:55.574003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438236+0000","last_change":"2023-07-14T18:10:17.214663+0000","last_active":"2023-07-14T18:10:23.438236+0000","last_peered":"2023-07-14T18:10:23.438236+0000","last_clean":"2023-07-14T18:10:23.438236+0000","last_became_active":"2023-07-14T18:10:17.214355+0000","last_became_peered":"2023-07-14T18:10:17.214355+0000","last_unstale":"2023-07-14T18:10:23.438236+0000","last_undegraded":"2023-07-14T18:10:23.438236+0000","last_fullsized":"2023-07-14T18:10:23.438236+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:56.306784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088750+0000","last_change":"2023-07-14T18:10:18.124903+0000","last_active":"2023-07-14T18:10:24.088750+0000","last_peered":"2023-07-14T18:10:24.088750+0000","last_clean":"2023-07-14T18:10:24.088750+0000","last_became_active":"2023-07-14T18:10:18.124661+0000","last_became_peered":"2023-07-14T18:10:18.124661+0000","last_unstale":"2023-07-14T18:10:24.088750+0000","last_undegraded":"2023-07-14T18:10:24.088750+0000","last_fullsized":"2023-07-14T18:10:24.088750+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:21.959726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846209+0000","last_change":"2023-07-14T18:10:22.459746+0000","last_active":"2023-07-14T18:10:23.846209+0000","last_peered":"2023-07-14T18:10:23.846209+0000","last_clean":"2023-07-14T18:10:23.846209+0000","last_became_active":"2023-07-14T18:10:22.459331+0000","last_became_peered":"2023-07-14T18:10:22.459331+0000","last_unstale":"2023-07-14T18:10:23.846209+0000","last_undegraded":"2023-07-14T18:10:23.846209+0000","last_fullsized":"2023-07-14T18:10:23.846209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:34.082947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430093+0000","last_change":"2023-07-14T18:10:17.213362+0000","last_active":"2023-07-14T18:10:37.430093+0000","last_peered":"2023-07-14T18:10:37.430093+0000","last_clean":"2023-07-14T18:10:37.430093+0000","last_became_active":"2023-07-14T18:10:17.212914+0000","last_became_peered":"2023-07-14T18:10:17.212914+0000","last_unstale":"2023-07-14T18:10:37.430093+0000","last_undegraded":"2023-07-14T18:10:37.430093+0000","last_fullsized":"2023-07-14T18:10:37.430093+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:08.051946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090127+0000","last_change":"2023-07-14T18:10:18.106583+0000","last_active":"2023-07-14T18:10:24.090127+0000","last_peered":"2023-07-14T18:10:24.090127+0000","last_clean":"2023-07-14T18:10:24.090127+0000","last_became_active":"2023-07-14T18:10:18.106216+0000","last_became_peered":"2023-07-14T18:10:18.106216+0000","last_unstale":"2023-07-14T18:10:24.090127+0000","last_undegraded":"2023-07-14T18:10:24.090127+0000","last_fullsized":"2023-07-14T18:10:24.090127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:03:32.095853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089940+0000","last_change":"2023-07-14T18:10:22.447133+0000","last_active":"2023-07-14T18:10:24.089940+0000","last_peered":"2023-07-14T18:10:24.089940+0000","last_clean":"2023-07-14T18:10:24.089940+0000","last_became_active":"2023-07-14T18:10:22.446716+0000","last_became_peered":"2023-07-14T18:10:22.446716+0000","last_unstale":"2023-07-14T18:10:24.089940+0000","last_undegraded":"2023-07-14T18:10:24.089940+0000","last_fullsized":"2023-07-14T18:10:24.089940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:49.677906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.439030+0000","last_change":"2023-07-14T18:10:17.210729+0000","last_active":"2023-07-14T18:10:23.439030+0000","last_peered":"2023-07-14T18:10:23.439030+0000","last_clean":"2023-07-14T18:10:23.439030+0000","last_became_active":"2023-07-14T18:10:17.210449+0000","last_became_peered":"2023-07-14T18:10:17.210449+0000","last_unstale":"2023-07-14T18:10:23.439030+0000","last_undegraded":"2023-07-14T18:10:23.439030+0000","last_fullsized":"2023-07-14T18:10:23.439030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:57.909641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089476+0000","last_change":"2023-07-14T18:10:18.107118+0000","last_active":"2023-07-14T18:10:24.089476+0000","last_peered":"2023-07-14T18:10:24.089476+0000","last_clean":"2023-07-14T18:10:24.089476+0000","last_became_active":"2023-07-14T18:10:18.106367+0000","last_became_peered":"2023-07-14T18:10:18.106367+0000","last_unstale":"2023-07-14T18:10:24.089476+0000","last_undegraded":"2023-07-14T18:10:24.089476+0000","last_fullsized":"2023-07-14T18:10:24.089476+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:50:19.299272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438592+0000","last_change":"2023-07-14T18:10:22.500692+0000","last_active":"2023-07-14T18:10:23.438592+0000","last_peered":"2023-07-14T18:10:23.438592+0000","last_clean":"2023-07-14T18:10:23.438592+0000","last_became_active":"2023-07-14T18:10:22.500239+0000","last_became_peered":"2023-07-14T18:10:22.500239+0000","last_unstale":"2023-07-14T18:10:23.438592+0000","last_undegraded":"2023-07-14T18:10:23.438592+0000","last_fullsized":"2023-07-14T18:10:23.438592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:06.716562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089450+0000","last_change":"2023-07-14T18:10:17.212519+0000","last_active":"2023-07-14T18:10:24.089450+0000","last_peered":"2023-07-14T18:10:24.089450+0000","last_clean":"2023-07-14T18:10:24.089450+0000","last_became_active":"2023-07-14T18:10:17.212228+0000","last_became_peered":"2023-07-14T18:10:17.212228+0000","last_unstale":"2023-07-14T18:10:24.089450+0000","last_undegraded":"2023-07-14T18:10:24.089450+0000","last_fullsized":"2023-07-14T18:10:24.089450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:25:19.995310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734905+0000","last_change":"2023-07-14T18:10:18.111808+0000","last_active":"2023-07-14T18:10:23.734905+0000","last_peered":"2023-07-14T18:10:23.734905+0000","last_clean":"2023-07-14T18:10:23.734905+0000","last_became_active":"2023-07-14T18:10:18.111501+0000","last_became_peered":"2023-07-14T18:10:18.111501+0000","last_unstale":"2023-07-14T18:10:23.734905+0000","last_undegraded":"2023-07-14T18:10:23.734905+0000","last_fullsized":"2023-07-14T18:10:23.734905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:36.111363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845939+0000","last_change":"2023-07-14T18:10:22.453153+0000","last_active":"2023-07-14T18:10:23.845939+0000","last_peered":"2023-07-14T18:10:23.845939+0000","last_clean":"2023-07-14T18:10:23.845939+0000","last_became_active":"2023-07-14T18:10:22.452767+0000","last_became_peered":"2023-07-14T18:10:22.452767+0000","last_unstale":"2023-07-14T18:10:23.845939+0000","last_undegraded":"2023-07-14T18:10:23.845939+0000","last_fullsized":"2023-07-14T18:10:23.845939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:10:07.917749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089769+0000","last_change":"2023-07-14T18:10:17.215950+0000","last_active":"2023-07-14T18:10:24.089769+0000","last_peered":"2023-07-14T18:10:24.089769+0000","last_clean":"2023-07-14T18:10:24.089769+0000","last_became_active":"2023-07-14T18:10:17.215690+0000","last_became_peered":"2023-07-14T18:10:17.215690+0000","last_unstale":"2023-07-14T18:10:24.089769+0000","last_undegraded":"2023-07-14T18:10:24.089769+0000","last_fullsized":"2023-07-14T18:10:24.089769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:54.597911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734912+0000","last_change":"2023-07-14T18:10:18.114043+0000","last_active":"2023-07-14T18:10:23.734912+0000","last_peered":"2023-07-14T18:10:23.734912+0000","last_clean":"2023-07-14T18:10:23.734912+0000","last_became_active":"2023-07-14T18:10:18.113745+0000","last_became_peered":"2023-07-14T18:10:18.113745+0000","last_unstale":"2023-07-14T18:10:23.734912+0000","last_undegraded":"2023-07-14T18:10:23.734912+0000","last_fullsized":"2023-07-14T18:10:23.734912+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:42:18.691206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846054+0000","last_change":"2023-07-14T18:10:22.501904+0000","last_active":"2023-07-14T18:10:23.846054+0000","last_peered":"2023-07-14T18:10:23.846054+0000","last_clean":"2023-07-14T18:10:23.846054+0000","last_became_active":"2023-07-14T18:10:22.501500+0000","last_became_peered":"2023-07-14T18:10:22.501500+0000","last_unstale":"2023-07-14T18:10:23.846054+0000","last_undegraded":"2023-07-14T18:10:23.846054+0000","last_fullsized":"2023-07-14T18:10:23.846054+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:27:26.164775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090043+0000","last_change":"2023-07-14T18:10:18.114655+0000","last_active":"2023-07-14T18:10:24.090043+0000","last_peered":"2023-07-14T18:10:24.090043+0000","last_clean":"2023-07-14T18:10:24.090043+0000","last_became_active":"2023-07-14T18:10:18.113984+0000","last_became_peered":"2023-07-14T18:10:18.113984+0000","last_unstale":"2023-07-14T18:10:24.090043+0000","last_undegraded":"2023-07-14T18:10:24.090043+0000","last_fullsized":"2023-07-14T18:10:24.090043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:51:36.108289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430903+0000","last_change":"2023-07-14T18:10:17.212612+0000","last_active":"2023-07-14T18:10:37.430903+0000","last_peered":"2023-07-14T18:10:37.430903+0000","last_clean":"2023-07-14T18:10:37.430903+0000","last_became_active":"2023-07-14T18:10:17.212282+0000","last_became_peered":"2023-07-14T18:10:17.212282+0000","last_unstale":"2023-07-14T18:10:37.430903+0000","last_undegraded":"2023-07-14T18:10:37.430903+0000","last_fullsized":"2023-07-14T18:10:37.430903+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:12:46.994247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089078+0000","last_change":"2023-07-14T18:10:22.449831+0000","last_active":"2023-07-14T18:10:24.089078+0000","last_peered":"2023-07-14T18:10:24.089078+0000","last_clean":"2023-07-14T18:10:24.089078+0000","last_became_active":"2023-07-14T18:10:22.449380+0000","last_became_peered":"2023-07-14T18:10:22.449380+0000","last_unstale":"2023-07-14T18:10:24.089078+0000","last_undegraded":"2023-07-14T18:10:24.089078+0000","last_fullsized":"2023-07-14T18:10:24.089078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:33.180987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088819+0000","last_change":"2023-07-14T18:10:18.107041+0000","last_active":"2023-07-14T18:10:24.088819+0000","last_peered":"2023-07-14T18:10:24.088819+0000","last_clean":"2023-07-14T18:10:24.088819+0000","last_became_active":"2023-07-14T18:10:18.106715+0000","last_became_peered":"2023-07-14T18:10:18.106715+0000","last_unstale":"2023-07-14T18:10:24.088819+0000","last_undegraded":"2023-07-14T18:10:24.088819+0000","last_fullsized":"2023-07-14T18:10:24.088819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:49.905307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438471+0000","last_change":"2023-07-14T18:10:17.205525+0000","last_active":"2023-07-14T18:10:23.438471+0000","last_peered":"2023-07-14T18:10:23.438471+0000","last_clean":"2023-07-14T18:10:23.438471+0000","last_became_active":"2023-07-14T18:10:17.205164+0000","last_became_peered":"2023-07-14T18:10:17.205164+0000","last_unstale":"2023-07-14T18:10:23.438471+0000","last_undegraded":"2023-07-14T18:10:23.438471+0000","last_fullsized":"2023-07-14T18:10:23.438471+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:06:06.720104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088564+0000","last_change":"2023-07-14T18:09:18.495776+0000","last_active":"2023-07-14T18:10:24.088564+0000","last_peered":"2023-07-14T18:10:24.088564+0000","last_clean":"2023-07-14T18:10:24.088564+0000","last_became_active":"2023-07-14T18:09:18.495326+0000","last_became_peered":"2023-07-14T18:09:18.495326+0000","last_unstale":"2023-07-14T18:10:24.088564+0000","last_undegraded":"2023-07-14T18:10:24.088564+0000","last_fullsized":"2023-07-14T18:10:24.088564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:06:31.483975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:06:31.483975+0000","last_clean_scrub_stamp":"2023-07-14T18:06:31.483975+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:42:18.539340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090230+0000","last_change":"2023-07-14T18:10:22.501067+0000","last_active":"2023-07-14T18:10:24.090230+0000","last_peered":"2023-07-14T18:10:24.090230+0000","last_clean":"2023-07-14T18:10:24.090230+0000","last_became_active":"2023-07-14T18:10:22.500337+0000","last_became_peered":"2023-07-14T18:10:22.500337+0000","last_unstale":"2023-07-14T18:10:24.090230+0000","last_undegraded":"2023-07-14T18:10:24.090230+0000","last_fullsized":"2023-07-14T18:10:24.090230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:43.900787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845551+0000","last_change":"2023-07-14T18:10:18.107299+0000","last_active":"2023-07-14T18:10:23.845551+0000","last_peered":"2023-07-14T18:10:23.845551+0000","last_clean":"2023-07-14T18:10:23.845551+0000","last_became_active":"2023-07-14T18:10:18.106986+0000","last_became_peered":"2023-07-14T18:10:18.106986+0000","last_unstale":"2023-07-14T18:10:23.845551+0000","last_undegraded":"2023-07-14T18:10:23.845551+0000","last_fullsized":"2023-07-14T18:10:23.845551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:47.211264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429208+0000","last_change":"2023-07-14T18:10:17.217895+0000","last_active":"2023-07-14T18:10:37.429208+0000","last_peered":"2023-07-14T18:10:37.429208+0000","last_clean":"2023-07-14T18:10:37.429208+0000","last_became_active":"2023-07-14T18:10:17.217548+0000","last_became_peered":"2023-07-14T18:10:17.217548+0000","last_unstale":"2023-07-14T18:10:37.429208+0000","last_undegraded":"2023-07-14T18:10:37.429208+0000","last_fullsized":"2023-07-14T18:10:37.429208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:29:08.392282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088848+0000","last_change":"2023-07-14T18:10:22.453140+0000","last_active":"2023-07-14T18:10:24.088848+0000","last_peered":"2023-07-14T18:10:24.088848+0000","last_clean":"2023-07-14T18:10:24.088848+0000","last_became_active":"2023-07-14T18:10:22.452731+0000","last_became_peered":"2023-07-14T18:10:22.452731+0000","last_unstale":"2023-07-14T18:10:24.088848+0000","last_undegraded":"2023-07-14T18:10:24.088848+0000","last_fullsized":"2023-07-14T18:10:24.088848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:40.973261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980328+0000","last_change":"2023-07-14T18:10:17.213431+0000","last_active":"2023-07-14T18:10:36.980328+0000","last_peered":"2023-07-14T18:10:36.980328+0000","last_clean":"2023-07-14T18:10:36.980328+0000","last_became_active":"2023-07-14T18:10:17.213125+0000","last_became_peered":"2023-07-14T18:10:17.213125+0000","last_unstale":"2023-07-14T18:10:36.980328+0000","last_undegraded":"2023-07-14T18:10:36.980328+0000","last_fullsized":"2023-07-14T18:10:36.980328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:53:36.345583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846032+0000","last_change":"2023-07-14T18:10:18.110593+0000","last_active":"2023-07-14T18:10:23.846032+0000","last_peered":"2023-07-14T18:10:23.846032+0000","last_clean":"2023-07-14T18:10:23.846032+0000","last_became_active":"2023-07-14T18:10:18.110220+0000","last_became_peered":"2023-07-14T18:10:18.110220+0000","last_unstale":"2023-07-14T18:10:23.846032+0000","last_undegraded":"2023-07-14T18:10:23.846032+0000","last_fullsized":"2023-07-14T18:10:23.846032+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:58:18.893324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088540+0000","last_change":"2023-07-14T18:10:22.500809+0000","last_active":"2023-07-14T18:10:24.088540+0000","last_peered":"2023-07-14T18:10:24.088540+0000","last_clean":"2023-07-14T18:10:24.088540+0000","last_became_active":"2023-07-14T18:10:22.500387+0000","last_became_peered":"2023-07-14T18:10:22.500387+0000","last_unstale":"2023-07-14T18:10:24.088540+0000","last_undegraded":"2023-07-14T18:10:24.088540+0000","last_fullsized":"2023-07-14T18:10:24.088540+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:26.241462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089871+0000","last_change":"2023-07-14T18:10:17.209632+0000","last_active":"2023-07-14T18:10:24.089871+0000","last_peered":"2023-07-14T18:10:24.089871+0000","last_clean":"2023-07-14T18:10:24.089871+0000","last_became_active":"2023-07-14T18:10:17.209274+0000","last_became_peered":"2023-07-14T18:10:17.209274+0000","last_unstale":"2023-07-14T18:10:24.089871+0000","last_undegraded":"2023-07-14T18:10:24.089871+0000","last_fullsized":"2023-07-14T18:10:24.089871+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:49:02.203177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089197+0000","last_change":"2023-07-14T18:10:18.119828+0000","last_active":"2023-07-14T18:10:24.089197+0000","last_peered":"2023-07-14T18:10:24.089197+0000","last_clean":"2023-07-14T18:10:24.089197+0000","last_became_active":"2023-07-14T18:10:18.119551+0000","last_became_peered":"2023-07-14T18:10:18.119551+0000","last_unstale":"2023-07-14T18:10:24.089197+0000","last_undegraded":"2023-07-14T18:10:24.089197+0000","last_fullsized":"2023-07-14T18:10:24.089197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:59:11.985338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549754+0000","last_change":"2023-07-14T18:10:22.450325+0000","last_active":"2023-07-14T18:10:23.549754+0000","last_peered":"2023-07-14T18:10:23.549754+0000","last_clean":"2023-07-14T18:10:23.549754+0000","last_became_active":"2023-07-14T18:10:22.449492+0000","last_became_peered":"2023-07-14T18:10:22.449492+0000","last_unstale":"2023-07-14T18:10:23.549754+0000","last_undegraded":"2023-07-14T18:10:23.549754+0000","last_fullsized":"2023-07-14T18:10:23.549754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:37:37.829472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438842+0000","last_change":"2023-07-14T18:10:17.212891+0000","last_active":"2023-07-14T18:10:23.438842+0000","last_peered":"2023-07-14T18:10:23.438842+0000","last_clean":"2023-07-14T18:10:23.438842+0000","last_became_active":"2023-07-14T18:10:17.212573+0000","last_became_peered":"2023-07-14T18:10:17.212573+0000","last_unstale":"2023-07-14T18:10:23.438842+0000","last_undegraded":"2023-07-14T18:10:23.438842+0000","last_fullsized":"2023-07-14T18:10:23.438842+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:38:07.247348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440729+0000","last_change":"2023-07-14T18:10:18.113389+0000","last_active":"2023-07-14T18:10:23.440729+0000","last_peered":"2023-07-14T18:10:23.440729+0000","last_clean":"2023-07-14T18:10:23.440729+0000","last_became_active":"2023-07-14T18:10:18.113053+0000","last_became_peered":"2023-07-14T18:10:18.113053+0000","last_unstale":"2023-07-14T18:10:23.440729+0000","last_undegraded":"2023-07-14T18:10:23.440729+0000","last_fullsized":"2023-07-14T18:10:23.440729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:29:31.077735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438597+0000","last_change":"2023-07-14T18:10:22.500749+0000","last_active":"2023-07-14T18:10:23.438597+0000","last_peered":"2023-07-14T18:10:23.438597+0000","last_clean":"2023-07-14T18:10:23.438597+0000","last_became_active":"2023-07-14T18:10:22.500357+0000","last_became_peered":"2023-07-14T18:10:22.500357+0000","last_unstale":"2023-07-14T18:10:23.438597+0000","last_undegraded":"2023-07-14T18:10:23.438597+0000","last_fullsized":"2023-07-14T18:10:23.438597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:05:10.947498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980089+0000","last_change":"2023-07-14T18:10:17.212501+0000","last_active":"2023-07-14T18:10:36.980089+0000","last_peered":"2023-07-14T18:10:36.980089+0000","last_clean":"2023-07-14T18:10:36.980089+0000","last_became_active":"2023-07-14T18:10:17.212171+0000","last_became_peered":"2023-07-14T18:10:17.212171+0000","last_unstale":"2023-07-14T18:10:36.980089+0000","last_undegraded":"2023-07-14T18:10:36.980089+0000","last_fullsized":"2023-07-14T18:10:36.980089+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:37.208644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438420+0000","last_change":"2023-07-14T18:10:18.111692+0000","last_active":"2023-07-14T18:10:23.438420+0000","last_peered":"2023-07-14T18:10:23.438420+0000","last_clean":"2023-07-14T18:10:23.438420+0000","last_became_active":"2023-07-14T18:10:18.111424+0000","last_became_peered":"2023-07-14T18:10:18.111424+0000","last_unstale":"2023-07-14T18:10:23.438420+0000","last_undegraded":"2023-07-14T18:10:23.438420+0000","last_fullsized":"2023-07-14T18:10:23.438420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:06.610911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089989+0000","last_change":"2023-07-14T18:10:22.456773+0000","last_active":"2023-07-14T18:10:24.089989+0000","last_peered":"2023-07-14T18:10:24.089989+0000","last_clean":"2023-07-14T18:10:24.089989+0000","last_became_active":"2023-07-14T18:10:22.456309+0000","last_became_peered":"2023-07-14T18:10:22.456309+0000","last_unstale":"2023-07-14T18:10:24.089989+0000","last_undegraded":"2023-07-14T18:10:24.089989+0000","last_fullsized":"2023-07-14T18:10:24.089989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:29:09.644386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088932+0000","last_change":"2023-07-14T18:10:17.205109+0000","last_active":"2023-07-14T18:10:24.088932+0000","last_peered":"2023-07-14T18:10:24.088932+0000","last_clean":"2023-07-14T18:10:24.088932+0000","last_became_active":"2023-07-14T18:10:17.204644+0000","last_became_peered":"2023-07-14T18:10:17.204644+0000","last_unstale":"2023-07-14T18:10:24.088932+0000","last_undegraded":"2023-07-14T18:10:24.088932+0000","last_fullsized":"2023-07-14T18:10:24.088932+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:08.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088692+0000","last_change":"2023-07-14T18:10:18.109143+0000","last_active":"2023-07-14T18:10:24.088692+0000","last_peered":"2023-07-14T18:10:24.088692+0000","last_clean":"2023-07-14T18:10:24.088692+0000","last_became_active":"2023-07-14T18:10:18.108841+0000","last_became_peered":"2023-07-14T18:10:18.108841+0000","last_unstale":"2023-07-14T18:10:24.088692+0000","last_undegraded":"2023-07-14T18:10:24.088692+0000","last_fullsized":"2023-07-14T18:10:24.088692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:30:16.212954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438316+0000","last_change":"2023-07-14T18:10:22.450564+0000","last_active":"2023-07-14T18:10:23.438316+0000","last_peered":"2023-07-14T18:10:23.438316+0000","last_clean":"2023-07-14T18:10:23.438316+0000","last_became_active":"2023-07-14T18:10:22.449801+0000","last_became_peered":"2023-07-14T18:10:22.449801+0000","last_unstale":"2023-07-14T18:10:23.438316+0000","last_undegraded":"2023-07-14T18:10:23.438316+0000","last_fullsized":"2023-07-14T18:10:23.438316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:00:04.711449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549788+0000","last_change":"2023-07-14T18:10:17.211522+0000","last_active":"2023-07-14T18:10:23.549788+0000","last_peered":"2023-07-14T18:10:23.549788+0000","last_clean":"2023-07-14T18:10:23.549788+0000","last_became_active":"2023-07-14T18:10:17.211163+0000","last_became_peered":"2023-07-14T18:10:17.211163+0000","last_unstale":"2023-07-14T18:10:23.549788+0000","last_undegraded":"2023-07-14T18:10:23.549788+0000","last_fullsized":"2023-07-14T18:10:23.549788+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:14:47.230105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089155+0000","last_change":"2023-07-14T18:10:18.111885+0000","last_active":"2023-07-14T18:10:24.089155+0000","last_peered":"2023-07-14T18:10:24.089155+0000","last_clean":"2023-07-14T18:10:24.089155+0000","last_became_active":"2023-07-14T18:10:18.111520+0000","last_became_peered":"2023-07-14T18:10:18.111520+0000","last_unstale":"2023-07-14T18:10:24.089155+0000","last_undegraded":"2023-07-14T18:10:24.089155+0000","last_fullsized":"2023-07-14T18:10:24.089155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:06:29.717829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438617+0000","last_change":"2023-07-14T18:10:22.500612+0000","last_active":"2023-07-14T18:10:23.438617+0000","last_peered":"2023-07-14T18:10:23.438617+0000","last_clean":"2023-07-14T18:10:23.438617+0000","last_became_active":"2023-07-14T18:10:22.500219+0000","last_became_peered":"2023-07-14T18:10:22.500219+0000","last_unstale":"2023-07-14T18:10:23.438617+0000","last_undegraded":"2023-07-14T18:10:23.438617+0000","last_fullsized":"2023-07-14T18:10:23.438617+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:09.251191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981056+0000","last_change":"2023-07-14T18:10:17.214307+0000","last_active":"2023-07-14T18:10:36.981056+0000","last_peered":"2023-07-14T18:10:36.981056+0000","last_clean":"2023-07-14T18:10:36.981056+0000","last_became_active":"2023-07-14T18:10:17.214004+0000","last_became_peered":"2023-07-14T18:10:17.214004+0000","last_unstale":"2023-07-14T18:10:36.981056+0000","last_undegraded":"2023-07-14T18:10:36.981056+0000","last_fullsized":"2023-07-14T18:10:36.981056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:50:59.158356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089823+0000","last_change":"2023-07-14T18:10:18.120504+0000","last_active":"2023-07-14T18:10:24.089823+0000","last_peered":"2023-07-14T18:10:24.089823+0000","last_clean":"2023-07-14T18:10:24.089823+0000","last_became_active":"2023-07-14T18:10:18.120133+0000","last_became_peered":"2023-07-14T18:10:18.120133+0000","last_unstale":"2023-07-14T18:10:24.089823+0000","last_undegraded":"2023-07-14T18:10:24.089823+0000","last_fullsized":"2023-07-14T18:10:24.089823+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:54.324914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089420+0000","last_change":"2023-07-14T18:10:22.453040+0000","last_active":"2023-07-14T18:10:24.089420+0000","last_peered":"2023-07-14T18:10:24.089420+0000","last_clean":"2023-07-14T18:10:24.089420+0000","last_became_active":"2023-07-14T18:10:22.452694+0000","last_became_peered":"2023-07-14T18:10:22.452694+0000","last_unstale":"2023-07-14T18:10:24.089420+0000","last_undegraded":"2023-07-14T18:10:24.089420+0000","last_fullsized":"2023-07-14T18:10:24.089420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:43:06.299843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428686+0000","last_change":"2023-07-14T18:10:17.212480+0000","last_active":"2023-07-14T18:10:37.428686+0000","last_peered":"2023-07-14T18:10:37.428686+0000","last_clean":"2023-07-14T18:10:37.428686+0000","last_became_active":"2023-07-14T18:10:17.212080+0000","last_became_peered":"2023-07-14T18:10:17.212080+0000","last_unstale":"2023-07-14T18:10:37.428686+0000","last_undegraded":"2023-07-14T18:10:37.428686+0000","last_fullsized":"2023-07-14T18:10:37.428686+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:19:44.529873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089506+0000","last_change":"2023-07-14T18:10:18.121738+0000","last_active":"2023-07-14T18:10:24.089506+0000","last_peered":"2023-07-14T18:10:24.089506+0000","last_clean":"2023-07-14T18:10:24.089506+0000","last_became_active":"2023-07-14T18:10:18.121186+0000","last_became_peered":"2023-07-14T18:10:18.121186+0000","last_unstale":"2023-07-14T18:10:24.089506+0000","last_undegraded":"2023-07-14T18:10:24.089506+0000","last_fullsized":"2023-07-14T18:10:24.089506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:40:16.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440225+0000","last_change":"2023-07-14T18:10:22.461051+0000","last_active":"2023-07-14T18:10:23.440225+0000","last_peered":"2023-07-14T18:10:23.440225+0000","last_clean":"2023-07-14T18:10:23.440225+0000","last_became_active":"2023-07-14T18:10:22.460663+0000","last_became_peered":"2023-07-14T18:10:22.460663+0000","last_unstale":"2023-07-14T18:10:23.440225+0000","last_undegraded":"2023-07-14T18:10:23.440225+0000","last_fullsized":"2023-07-14T18:10:23.440225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:58:08.355441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090601+0000","last_change":"2023-07-14T18:10:17.211686+0000","last_active":"2023-07-14T18:10:24.090601+0000","last_peered":"2023-07-14T18:10:24.090601+0000","last_clean":"2023-07-14T18:10:24.090601+0000","last_became_active":"2023-07-14T18:10:17.211367+0000","last_became_peered":"2023-07-14T18:10:17.211367+0000","last_unstale":"2023-07-14T18:10:24.090601+0000","last_undegraded":"2023-07-14T18:10:24.090601+0000","last_fullsized":"2023-07-14T18:10:24.090601+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:50.829623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089608+0000","last_change":"2023-07-14T18:10:18.116960+0000","last_active":"2023-07-14T18:10:24.089608+0000","last_peered":"2023-07-14T18:10:24.089608+0000","last_clean":"2023-07-14T18:10:24.089608+0000","last_became_active":"2023-07-14T18:10:18.116416+0000","last_became_peered":"2023-07-14T18:10:18.116416+0000","last_unstale":"2023-07-14T18:10:24.089608+0000","last_undegraded":"2023-07-14T18:10:24.089608+0000","last_fullsized":"2023-07-14T18:10:24.089608+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:33.066056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089552+0000","last_change":"2023-07-14T18:10:22.450045+0000","last_active":"2023-07-14T18:10:24.089552+0000","last_peered":"2023-07-14T18:10:24.089552+0000","last_clean":"2023-07-14T18:10:24.089552+0000","last_became_active":"2023-07-14T18:10:22.449621+0000","last_became_peered":"2023-07-14T18:10:22.449621+0000","last_unstale":"2023-07-14T18:10:24.089552+0000","last_undegraded":"2023-07-14T18:10:24.089552+0000","last_fullsized":"2023-07-14T18:10:24.089552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:48:23.837464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846188+0000","last_change":"2023-07-14T18:10:17.210691+0000","last_active":"2023-07-14T18:10:23.846188+0000","last_peered":"2023-07-14T18:10:23.846188+0000","last_clean":"2023-07-14T18:10:23.846188+0000","last_became_active":"2023-07-14T18:10:17.210391+0000","last_became_peered":"2023-07-14T18:10:17.210391+0000","last_unstale":"2023-07-14T18:10:23.846188+0000","last_undegraded":"2023-07-14T18:10:23.846188+0000","last_fullsized":"2023-07-14T18:10:23.846188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:20:04.361243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089271+0000","last_change":"2023-07-14T18:10:18.115347+0000","last_active":"2023-07-14T18:10:24.089271+0000","last_peered":"2023-07-14T18:10:24.089271+0000","last_clean":"2023-07-14T18:10:24.089271+0000","last_became_active":"2023-07-14T18:10:18.115027+0000","last_became_peered":"2023-07-14T18:10:18.115027+0000","last_unstale":"2023-07-14T18:10:24.089271+0000","last_undegraded":"2023-07-14T18:10:24.089271+0000","last_fullsized":"2023-07-14T18:10:24.089271+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:33:02.993257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088765+0000","last_change":"2023-07-14T18:10:18.103185+0000","last_active":"2023-07-14T18:10:24.088765+0000","last_peered":"2023-07-14T18:10:24.088765+0000","last_clean":"2023-07-14T18:10:24.088765+0000","last_became_active":"2023-07-14T18:10:18.102800+0000","last_became_peered":"2023-07-14T18:10:18.102800+0000","last_unstale":"2023-07-14T18:10:24.088765+0000","last_undegraded":"2023-07-14T18:10:24.088765+0000","last_fullsized":"2023-07-14T18:10:24.088765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:35:22.321627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550057+0000","last_change":"2023-07-14T18:10:18.113501+0000","last_active":"2023-07-14T18:10:23.550057+0000","last_peered":"2023-07-14T18:10:23.550057+0000","last_clean":"2023-07-14T18:10:23.550057+0000","last_became_active":"2023-07-14T18:10:18.113205+0000","last_became_peered":"2023-07-14T18:10:18.113205+0000","last_unstale":"2023-07-14T18:10:23.550057+0000","last_undegraded":"2023-07-14T18:10:23.550057+0000","last_fullsized":"2023-07-14T18:10:23.550057+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:13:56.420803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549722+0000","last_change":"2023-07-14T18:10:18.114282+0000","last_active":"2023-07-14T18:10:23.549722+0000","last_peered":"2023-07-14T18:10:23.549722+0000","last_clean":"2023-07-14T18:10:23.549722+0000","last_became_active":"2023-07-14T18:10:18.113968+0000","last_became_peered":"2023-07-14T18:10:18.113968+0000","last_unstale":"2023-07-14T18:10:23.549722+0000","last_undegraded":"2023-07-14T18:10:23.549722+0000","last_fullsized":"2023-07-14T18:10:23.549722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:36.356073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440436+0000","last_change":"2023-07-14T18:10:18.116781+0000","last_active":"2023-07-14T18:10:23.440436+0000","last_peered":"2023-07-14T18:10:23.440436+0000","last_clean":"2023-07-14T18:10:23.440436+0000","last_became_active":"2023-07-14T18:10:18.116352+0000","last_became_peered":"2023-07-14T18:10:18.116352+0000","last_unstale":"2023-07-14T18:10:23.440436+0000","last_undegraded":"2023-07-14T18:10:23.440436+0000","last_fullsized":"2023-07-14T18:10:23.440436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:48:03.670475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846088+0000","last_change":"2023-07-14T18:10:18.112291+0000","last_active":"2023-07-14T18:10:23.846088+0000","last_peered":"2023-07-14T18:10:23.846088+0000","last_clean":"2023-07-14T18:10:23.846088+0000","last_became_active":"2023-07-14T18:10:18.111989+0000","last_became_peered":"2023-07-14T18:10:18.111989+0000","last_unstale":"2023-07-14T18:10:23.846088+0000","last_undegraded":"2023-07-14T18:10:23.846088+0000","last_fullsized":"2023-07-14T18:10:23.846088+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:11:33.924006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089268+0000","last_change":"2023-07-14T18:10:18.112316+0000","last_active":"2023-07-14T18:10:24.089268+0000","last_peered":"2023-07-14T18:10:24.089268+0000","last_clean":"2023-07-14T18:10:24.089268+0000","last_became_active":"2023-07-14T18:10:18.112030+0000","last_became_peered":"2023-07-14T18:10:18.112030+0000","last_unstale":"2023-07-14T18:10:24.089268+0000","last_undegraded":"2023-07-14T18:10:24.089268+0000","last_fullsized":"2023-07-14T18:10:24.089268+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:14:55.405097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550228+0000","last_change":"2023-07-14T18:10:18.116343+0000","last_active":"2023-07-14T18:10:23.550228+0000","last_peered":"2023-07-14T18:10:23.550228+0000","last_clean":"2023-07-14T18:10:23.550228+0000","last_became_active":"2023-07-14T18:10:18.116041+0000","last_became_peered":"2023-07-14T18:10:18.116041+0000","last_unstale":"2023-07-14T18:10:23.550228+0000","last_undegraded":"2023-07-14T18:10:23.550228+0000","last_fullsized":"2023-07-14T18:10:23.550228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:43:43.627634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089357+0000","last_change":"2023-07-14T18:10:18.117580+0000","last_active":"2023-07-14T18:10:24.089357+0000","last_peered":"2023-07-14T18:10:24.089357+0000","last_clean":"2023-07-14T18:10:24.089357+0000","last_became_active":"2023-07-14T18:10:18.117249+0000","last_became_peered":"2023-07-14T18:10:18.117249+0000","last_unstale":"2023-07-14T18:10:24.089357+0000","last_undegraded":"2023-07-14T18:10:24.089357+0000","last_fullsized":"2023-07-14T18:10:24.089357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:42.562595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089822+0000","last_change":"2023-07-14T18:10:18.107276+0000","last_active":"2023-07-14T18:10:24.089822+0000","last_peered":"2023-07-14T18:10:24.089822+0000","last_clean":"2023-07-14T18:10:24.089822+0000","last_became_active":"2023-07-14T18:10:18.107005+0000","last_became_peered":"2023-07-14T18:10:18.107005+0000","last_unstale":"2023-07-14T18:10:24.089822+0000","last_undegraded":"2023-07-14T18:10:24.089822+0000","last_fullsized":"2023-07-14T18:10:24.089822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:13.352444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089145+0000","last_change":"2023-07-14T18:10:18.107921+0000","last_active":"2023-07-14T18:10:24.089145+0000","last_peered":"2023-07-14T18:10:24.089145+0000","last_clean":"2023-07-14T18:10:24.089145+0000","last_became_active":"2023-07-14T18:10:18.107637+0000","last_became_peered":"2023-07-14T18:10:18.107637+0000","last_unstale":"2023-07-14T18:10:24.089145+0000","last_undegraded":"2023-07-14T18:10:24.089145+0000","last_fullsized":"2023-07-14T18:10:24.089145+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:32.687817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089709+0000","last_change":"2023-07-14T18:10:18.120125+0000","last_active":"2023-07-14T18:10:24.089709+0000","last_peered":"2023-07-14T18:10:24.089709+0000","last_clean":"2023-07-14T18:10:24.089709+0000","last_became_active":"2023-07-14T18:10:18.119814+0000","last_became_peered":"2023-07-14T18:10:18.119814+0000","last_unstale":"2023-07-14T18:10:24.089709+0000","last_undegraded":"2023-07-14T18:10:24.089709+0000","last_fullsized":"2023-07-14T18:10:24.089709+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:37:22.531320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845885+0000","last_change":"2023-07-14T18:10:18.113155+0000","last_active":"2023-07-14T18:10:23.845885+0000","last_peered":"2023-07-14T18:10:23.845885+0000","last_clean":"2023-07-14T18:10:23.845885+0000","last_became_active":"2023-07-14T18:10:18.112837+0000","last_became_peered":"2023-07-14T18:10:18.112837+0000","last_unstale":"2023-07-14T18:10:23.845885+0000","last_undegraded":"2023-07-14T18:10:23.845885+0000","last_fullsized":"2023-07-14T18:10:23.845885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:56:13.124803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088936+0000","last_change":"2023-07-14T18:10:18.112290+0000","last_active":"2023-07-14T18:10:24.088936+0000","last_peered":"2023-07-14T18:10:24.088936+0000","last_clean":"2023-07-14T18:10:24.088936+0000","last_became_active":"2023-07-14T18:10:18.111988+0000","last_became_peered":"2023-07-14T18:10:18.111988+0000","last_unstale":"2023-07-14T18:10:24.088936+0000","last_undegraded":"2023-07-14T18:10:24.088936+0000","last_fullsized":"2023-07-14T18:10:24.088936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:26.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6554,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":96,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":88042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449102,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":466382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":10,"up_from":71,"seq":304942678034,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1059825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.67500000000000004}]},{"osd":2,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.69099999999999995}]},{"osd":3,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.74199999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.80700000000000005}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.69299999999999995}]},{"osd":8,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.95799999999999996}]},{"osd":9,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.79500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]}]},{"osd":9,"up_from":66,"seq":283467841559,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":466382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.498}]},{"osd":2,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45300000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.37},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.46100000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.441}]},{"osd":5,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.59799999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.53300000000000003}]},{"osd":10,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.54400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":8,"up_from":58,"seq":249108103195,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":466207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.41499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.80500000000000005}]},{"osd":1,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.72199999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.42999999999999999,"15min":0.42299999999999999},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46899999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.499}]},{"osd":3,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.433,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79700000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.66800000000000004}]},{"osd":4,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64100000000000001,"15min":0.63900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.58499999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.36299999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.53900000000000003}]},{"osd":6,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.52300000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72099999999999997,"15min":0.72599999999999998},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.78600000000000003}]},{"osd":7,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.53100000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.70099999999999996,"15min":0.69299999999999995},"min":{"1min":0.501,"5min":0.439,"15min":0.439},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.85999999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.873}]},{"osd":10,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.91400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364831,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1057864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.41399999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.41399999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.57699999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.432,"15min":0.434},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.623,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.35199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.754,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58499999999999996,"15min":0.58099999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.84499999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54600000000000004,"15min":0.53800000000000003},"min":{"1min":0.432,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58199999999999996,"15min":0.57799999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61899999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.50700000000000001,"15min":0.50800000000000001},"min":{"1min":0.33800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.627,"15min":0.622},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82299999999999995,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.877},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67500000000000004,"15min":0.67700000000000005},"min":{"1min":0.46400000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.90600000000000003,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Jul 14 18:10:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.81799999999999995,"5min":1.024,"15min":1.024},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65000000000000002,"15min":0.64300000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.86499999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.78400000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:10:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.66700000000000004,"15min":0.67300000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.877,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.86099999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.71999999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.93300000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528355,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.311},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.29299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.34999999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76300000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.373}]},{"osd":2,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.374,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.442},{"interface":"front","average":{"1min":0.628,"5min":0.62,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.32200000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.42199999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.55700000000000005,"15min":0.54900000000000004},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.61399999999999999,"15min":0.60899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.59399999999999997}]},{"osd":5,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.51500000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:10:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.44600000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.94299999999999995,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.51100000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.57499999999999996}]},{"osd":8,"last update":"Fri Jul 14 18:09:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.628},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.495}]},{"osd":9,"last update":"Fri Jul 14 18:09:56 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54200000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:10:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":504315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.50900000000000001,"15min":0.48299999999999998},"min":{"1min":0.32800000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.51300000000000001,"15min":0.49099999999999999},"min":{"1min":0.36199999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.622,"15min":0.60399999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.97799999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.45300000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80600000000000005,"5min":1.337,"15min":1.337},"last":0.71999999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.54600000000000004,"15min":0.51600000000000001},"min":{"1min":0.41399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.43099999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.56799999999999995}]},{"osd":4,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.436,"15min":0.41899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45100000000000001,"15min":0.433},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.65800000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.47199999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92900000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.64300000000000002,"15min":0.62},"min":{"1min":0.45800000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.86599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.86599999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.503,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.48299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89000000000000001,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.79000000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59399999999999997,"15min":0.57999999999999996},"min":{"1min":0.38100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58499999999999996,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.97499999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70899999999999996,"15min":0.69599999999999995},"min":{"1min":0.59399999999999997,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66100000000000003,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.85299999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.92100000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:10:17 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.93600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]}]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":962560,"data_stored":465890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51400000000000001,"15min":0.49099999999999999},"min":{"1min":0.41099999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.51400000000000001,"15min":0.5},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.32000000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:09:32 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52000000000000002,"15min":0.51100000000000001},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78800000000000003,"5min":0.997,"15min":0.997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.52000000000000002,"15min":0.502},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri Jul 14 18:08:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.39800000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.46400000000000002,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54800000000000004,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.42999999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70999999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:09:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.44700000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.64200000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.34100000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.55500000000000005,"15min":0.54300000000000004},"min":{"1min":0.37,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.60399999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.65700000000000003,"15min":0.65900000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.92100000000000004,"5min":0.998,"15min":0.998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85299999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.51300000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:09:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.69299999999999995}]},{"osd":9,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.63100000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.54600000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":506007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.67500000000000004,"5min":0.995,"15min":0.995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68899999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.54000000000000004}]},{"osd":1,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.39000000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":1.29,"15min":1.29},"last":0.749},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.34899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.01,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76900000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:09:57 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55800000000000005,"15min":0.52800000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.94299999999999995,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.57199999999999995,"15min":0.54800000000000004},"min":{"1min":0.41099999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84899999999999998,"5min":0.996,"15min":0.996},"last":0.63900000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.54800000000000004,"15min":0.54300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55400000000000005,"15min":0.54400000000000004},"min":{"1min":0.45000000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.69399999999999995}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.498,"15min":0.47499999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.49199999999999999,"15min":0.46400000000000002},"min":{"1min":0.29699999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.82699999999999996,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:09:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.42999999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64100000000000001,"15min":0.63500000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95499999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.871}]},{"osd":7,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.92700000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67400000000000004,"15min":0.66700000000000004},"min":{"1min":0.40699999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":0.89100000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":1.032}]},{"osd":9,"last update":"Fri Jul 14 18:09:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.85499999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.0589999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":3,"up_from":25,"seq":107374182447,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":465573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.33900000000000002,"15min":0.32400000000000001},"min":{"1min":0.27100000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.36499999999999999,"15min":0.34499999999999997},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.30199999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54400000000000004,"15min":0.53200000000000003},"min":{"1min":0.375,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.53400000000000003},"min":{"1min":0.42099999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.36099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.58999999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.55500000000000005,"15min":0.52300000000000002},"min":{"1min":0.41699999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85899999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.39900000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.52800000000000002}]},{"osd":5,"last update":"Fri Jul 14 18:10:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.626,"15min":0.61199999999999999},"min":{"1min":0.47199999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78300000000000003,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.59899999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.52000000000000002,"15min":0.499},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.46800000000000003}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81999999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.69799999999999995,"15min":0.68799999999999994},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.51700000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:09:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.64100000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:09:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.622}]},{"osd":10,"last update":"Fri Jul 14 18:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.505},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.55100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953516,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":962560,"data_stored":465822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78500000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.378,"15min":0.36099999999999999},"min":{"1min":0.28100000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.39800000000000002,"15min":0.38200000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.316}]},{"osd":2,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.58199999999999996,"15min":0.56599999999999995},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92200000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.58199999999999996,"15min":0.56499999999999995},"min":{"1min":0.40899999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.54400000000000004},"min":{"1min":0.44700000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78300000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.54000000000000004},"min":{"1min":0.34799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73399999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.49299999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.439,"5min":0.373,"15min":0.373},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64200000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.45000000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88300000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.432}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56299999999999994,"15min":0.55800000000000005},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.47199999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67600000000000005,"15min":0.67300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.87,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.71899999999999997,"15min":0.71299999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.60699999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.58999999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.63}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":5,"up_from":37,"seq":158913789992,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1581056,"data_stored":1057596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.38200000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.78000000000000003,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.39100000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78500000000000003,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54500000000000004}]},{"osd":1,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.78200000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.624},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.58499999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.70699999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60299999999999998,"15min":0.60399999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77700000000000002,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.66400000000000003}]},{"osd":4,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.81100000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.90100000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:09:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.629,"15min":0.63400000000000001},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.755},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65300000000000002,"15min":0.65400000000000003},"min":{"1min":0.501,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82999999999999996,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.85799999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.47799999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86599999999999999,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45600000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.91800000000000004}]},{"osd":8,"last update":"Fri Jul 14 18:09:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.79900000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:09:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.81499999999999995}]},{"osd":10,"last update":"Fri Jul 14 18:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.871}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":38515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":40207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:10:41.507 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-14T18:10:41.507 INFO:tasks.cephadm:Setting up client nodes... 2023-07-14T18:10:41.508 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-14T18:10:41.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:41 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3551278773' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:41.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:41 smithi115 ceph-mon[127451]: pgmap v263: 129 pgs: 129 active+clean; 583 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.6 KiB/s wr, 4 op/s 2023-07-14T18:10:41.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:41 smithi115 ceph-mon[127451]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:10:41.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:41 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3551278773' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:41.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:41 smithi150 ceph-mon[128853]: pgmap v263: 129 pgs: 129 active+clean; 583 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.6 KiB/s wr, 4 op/s 2023-07-14T18:10:41.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:41 smithi150 ceph-mon[128853]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:10:41.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:41 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3551278773' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:41.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:41 smithi187 ceph-mon[129317]: pgmap v263: 129 pgs: 129 active+clean; 583 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.6 KiB/s wr, 4 op/s 2023-07-14T18:10:41.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:41 smithi187 ceph-mon[129317]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:10:42.087 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:42.737 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:42 smithi115 ceph-mon[127451]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:10:42.737 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:42 smithi115 ceph-mon[127451]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T18:10:42.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:42 smithi150 ceph-mon[128853]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:10:42.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:42 smithi150 ceph-mon[128853]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T18:10:42.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:42 smithi187 ceph-mon[129317]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:10:42.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:42 smithi187 ceph-mon[129317]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T18:10:43.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.486+0000 7f6d0a995700 1 -- 172.21.15.115:0/1136438722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040ffde0 msgr2=0x7f6d04100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:43.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.486+0000 7f6d0a995700 1 --2- 172.21.15.115:0/1136438722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040ffde0 0x7f6d04100230 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009990 tx=0x7f6cf8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 -- 172.21.15.115:0/1136438722 shutdown_connections 2023-07-14T18:10:43.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 --2- 172.21.15.115:0/1136438722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d04100770 0x7f6d04107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 --2- 172.21.15.115:0/1136438722 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d040febe0 0x7f6d040feff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 --2- 172.21.15.115:0/1136438722 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040ffde0 0x7f6d04100230 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 -- 172.21.15.115:0/1136438722 >> 172.21.15.115:0/1136438722 conn(0x7f6d040fa210 msgr2=0x7f6d040fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:43.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 -- 172.21.15.115:0/1136438722 shutdown_connections 2023-07-14T18:10:43.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 -- 172.21.15.115:0/1136438722 wait complete. 2023-07-14T18:10:43.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 Processor -- start 2023-07-14T18:10:43.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 -- start start 2023-07-14T18:10:43.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040febe0 0x7f6d041958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:43.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.487+0000 7f6d0a995700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d040ffde0 0x7f6d04195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:43.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d0a995700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d04100770 0x7f6d0419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:43.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d0a995700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0419d480 con 0x7f6d040febe0 2023-07-14T18:10:43.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d0a995700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0419d5c0 con 0x7f6d040ffde0 2023-07-14T18:10:43.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d0a995700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0419d750 con 0x7f6d04100770 2023-07-14T18:10:43.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d03fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040febe0 0x7f6d041958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:43.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d03fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040febe0 0x7f6d041958f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53132/0 (socket says 172.21.15.115:53132) 2023-07-14T18:10:43.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d03fff700 1 -- 172.21.15.115:0/291346048 learned_addr learned my addr 172.21.15.115:0/291346048 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:43.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d08f32700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d04100770 0x7f6d0419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:43.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d03fff700 1 -- 172.21.15.115:0/291346048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d04100770 msgr2=0x7f6d0419ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:43.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d03fff700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d04100770 0x7f6d0419ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d03fff700 1 -- 172.21.15.115:0/291346048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d040ffde0 msgr2=0x7f6d04195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:43.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d03fff700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d040ffde0 0x7f6d04195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d03fff700 1 -- 172.21.15.115:0/291346048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf80095f0 con 0x7f6d040febe0 2023-07-14T18:10:43.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d03fff700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040febe0 0x7f6d041958f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400d9b0 tx=0x7f6cf400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:43.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d017fa700 1 -- 172.21.15.115:0/291346048 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4004020 con 0x7f6d040febe0 2023-07-14T18:10:43.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d041014d0 con 0x7f6d040febe0 2023-07-14T18:10:43.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d017fa700 1 -- 172.21.15.115:0/291346048 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6cf4009730 con 0x7f6d040febe0 2023-07-14T18:10:43.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d017fa700 1 -- 172.21.15.115:0/291346048 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40039c0 con 0x7f6d040febe0 2023-07-14T18:10:43.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.488+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d04101a20 con 0x7f6d040febe0 2023-07-14T18:10:43.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.489+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0410a250 con 0x7f6d040febe0 2023-07-14T18:10:43.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.490+0000 7f6d017fa700 1 -- 172.21.15.115:0/291346048 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f6cf40098a0 con 0x7f6d040febe0 2023-07-14T18:10:43.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.491+0000 7f6d017fa700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6cec0a11e0 0x7f6cec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:43.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.491+0000 7f6d037fe700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6cec0a11e0 0x7f6cec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:43.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.491+0000 7f6d017fa700 1 -- 172.21.15.115:0/291346048 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6cf40cc940 con 0x7f6d040febe0 2023-07-14T18:10:43.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.491+0000 7f6d037fe700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6cec0a11e0 0x7f6cec0a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80098e0 tx=0x7f6cf8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:43.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.495+0000 7f6d017fa700 1 -- 172.21.15.115:0/291346048 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6cf4095d40 con 0x7f6d040febe0 2023-07-14T18:10:43.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:43 smithi115 ceph-mon[127451]: pgmap v264: 129 pgs: 129 active+clean; 619 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2023-07-14T18:10:43.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.755+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f6d04106750 con 0x7f6d040febe0 2023-07-14T18:10:43.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.760+0000 7f6d017fa700 1 -- 172.21.15.115:0/291346048 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f6cf4099360 con 0x7f6d040febe0 2023-07-14T18:10:43.797 INFO:teuthology.orchestra.run.smithi115.stdout:[client.0] 2023-07-14T18:10:43.798 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQCjj7FkFukVLRAAIrcKP21TxhkkcXLHlwm0kg== 2023-07-14T18:10:43.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6cec0a11e0 msgr2=0x7f6cec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:43.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6cec0a11e0 0x7f6cec0a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80098e0 tx=0x7f6cf8000bc0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040febe0 msgr2=0x7f6d041958f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040febe0 0x7f6d041958f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400d9b0 tx=0x7f6cf400dcc0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 shutdown_connections 2023-07-14T18:10:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6cec0a11e0 0x7f6cec0a3690 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d04100770 0x7f6d0419ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6d040febe0 0x7f6d041958f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 --2- 172.21.15.115:0/291346048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d040ffde0 0x7f6d04195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 >> 172.21.15.115:0/291346048 conn(0x7f6d040fa210 msgr2=0x7f6d040fc500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 shutdown_connections 2023-07-14T18:10:43.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:43.762+0000 7f6d0a995700 1 -- 172.21.15.115:0/291346048 wait complete. 2023-07-14T18:10:43.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:43 smithi150 ceph-mon[128853]: pgmap v264: 129 pgs: 129 active+clean; 619 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2023-07-14T18:10:43.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:43 smithi187 ceph-mon[129317]: pgmap v264: 129 pgs: 129 active+clean; 619 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2023-07-14T18:10:44.239 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:10:44.239 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-14T18:10:44.240 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-14T18:10:44.280 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-14T18:10:44.281 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-14T18:10:44.282 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph mgr dump --format=json 2023-07-14T18:10:44.855 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:44 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/291346048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T18:10:44.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:44 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/291346048' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T18:10:44.869 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:44.870 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:44 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/291346048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T18:10:44.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:44 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/291346048' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T18:10:44.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:44 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/291346048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T18:10:44.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:44 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/291346048' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T18:10:45.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:45 smithi115 ceph-mon[127451]: pgmap v265: 129 pgs: 129 active+clean; 619 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2023-07-14T18:10:45.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:45 smithi150 ceph-mon[128853]: pgmap v265: 129 pgs: 129 active+clean; 619 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2023-07-14T18:10:45.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:45 smithi187 ceph-mon[129317]: pgmap v265: 129 pgs: 129 active+clean; 619 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2023-07-14T18:10:46.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.145+0000 7f87de2d6700 1 -- 172.21.15.115:0/1508549799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8101150 msgr2=0x7f87d810bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:46.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.145+0000 7f87de2d6700 1 --2- 172.21.15.115:0/1508549799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8101150 0x7f87d810bb40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f87d000b0d0 tx=0x7f87d000b3e0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.145+0000 7f87de2d6700 1 -- 172.21.15.115:0/1508549799 shutdown_connections 2023-07-14T18:10:46.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.145+0000 7f87de2d6700 1 --2- 172.21.15.115:0/1508549799 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d8101150 0x7f87d810bb40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.145+0000 7f87de2d6700 1 --2- 172.21.15.115:0/1508549799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87d80fef40 0x7f87d80ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.145+0000 7f87de2d6700 1 --2- 172.21.15.115:0/1508549799 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87d81007a0 0x7f87d8100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.145+0000 7f87de2d6700 1 -- 172.21.15.115:0/1508549799 >> 172.21.15.115:0/1508549799 conn(0x7f87d8075c90 msgr2=0x7f87d8076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:46.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.145+0000 7f87de2d6700 1 -- 172.21.15.115:0/1508549799 shutdown_connections 2023-07-14T18:10:46.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.145+0000 7f87de2d6700 1 -- 172.21.15.115:0/1508549799 wait complete. 2023-07-14T18:10:46.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87de2d6700 1 Processor -- start 2023-07-14T18:10:46.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87de2d6700 1 -- start start 2023-07-14T18:10:46.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87de2d6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87d80fef40 0x7f87d8101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:46.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87de2d6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d81007a0 0x7f87d8101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:46.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87de2d6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87d8101150 0x7f87d8102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:46.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87de2d6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d81a42f0 con 0x7f87d81007a0 2023-07-14T18:10:46.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87de2d6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d81a4460 con 0x7f87d8101150 2023-07-14T18:10:46.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87de2d6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d81a4620 con 0x7f87d80fef40 2023-07-14T18:10:46.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87d77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d81007a0 0x7f87d8101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:46.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87d77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d81007a0 0x7f87d8101ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53146/0 (socket says 172.21.15.115:53146) 2023-07-14T18:10:46.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87d77fe700 1 -- 172.21.15.115:0/2978593494 learned_addr learned my addr 172.21.15.115:0/2978593494 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:46.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87d7fff700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87d80fef40 0x7f87d8101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:46.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87dc873700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87d8101150 0x7f87d8102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:46.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87d77fe700 1 -- 172.21.15.115:0/2978593494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87d80fef40 msgr2=0x7f87d8101980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:46.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87d77fe700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87d80fef40 0x7f87d8101980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87d77fe700 1 -- 172.21.15.115:0/2978593494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87d8101150 msgr2=0x7f87d8102440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:46.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.146+0000 7f87d77fe700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87d8101150 0x7f87d8102440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.147+0000 7f87d77fe700 1 -- 172.21.15.115:0/2978593494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d0009d00 con 0x7f87d81007a0 2023-07-14T18:10:46.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.147+0000 7f87d77fe700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d81007a0 0x7f87d8101ec0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f87c800d950 tx=0x7f87c800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:46.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.147+0000 7f87d57fa700 1 -- 172.21.15.115:0/2978593494 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8009940 con 0x7f87d81007a0 2023-07-14T18:10:46.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.147+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d81a4900 con 0x7f87d81007a0 2023-07-14T18:10:46.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.147+0000 7f87d57fa700 1 -- 172.21.15.115:0/2978593494 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f87c8010460 con 0x7f87d81007a0 2023-07-14T18:10:46.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.147+0000 7f87d57fa700 1 -- 172.21.15.115:0/2978593494 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c800f620 con 0x7f87d81007a0 2023-07-14T18:10:46.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.147+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d81a4e00 con 0x7f87d81007a0 2023-07-14T18:10:46.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.148+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d804e340 con 0x7f87d81007a0 2023-07-14T18:10:46.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.149+0000 7f87d57fa700 1 -- 172.21.15.115:0/2978593494 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f87c80105d0 con 0x7f87d81007a0 2023-07-14T18:10:46.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.149+0000 7f87d57fa700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87c40a11e0 0x7f87c40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:46.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.149+0000 7f87d7fff700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87c40a11e0 0x7f87c40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:46.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.149+0000 7f87d57fa700 1 -- 172.21.15.115:0/2978593494 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f87c80ccae0 con 0x7f87d81007a0 2023-07-14T18:10:46.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.150+0000 7f87d7fff700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87c40a11e0 0x7f87c40a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f87c0005950 tx=0x7f87c000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:46.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.153+0000 7f87d57fa700 1 -- 172.21.15.115:0/2978593494 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f87c8095ee0 con 0x7f87d81007a0 2023-07-14T18:10:46.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.394+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f87d81a4f90 con 0x7f87d81007a0 2023-07-14T18:10:46.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.398+0000 7f87d57fa700 1 -- 172.21.15.115:0/2978593494 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260134 (secure 0 0 0) 0x7f87c8016090 con 0x7f87d81007a0 2023-07-14T18:10:46.438 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.401+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87c40a11e0 msgr2=0x7f87c40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.401+0000 7f87de2d6700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87c40a11e0 0x7f87c40a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f87c0005950 tx=0x7f87c000a300 comp rx=0 tx=0).stop 2023-07-14T18:10:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.401+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d81007a0 msgr2=0x7f87d8101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.401+0000 7f87de2d6700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d81007a0 0x7f87d8101ec0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f87c800d950 tx=0x7f87c800dc60 comp rx=0 tx=0).stop 2023-07-14T18:10:46.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.402+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 shutdown_connections 2023-07-14T18:10:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.402+0000 7f87de2d6700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f87c40a11e0 0x7f87c40a3690 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.402+0000 7f87de2d6700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87d8101150 0x7f87d8102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.402+0000 7f87de2d6700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87d80fef40 0x7f87d8101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.402+0000 7f87de2d6700 1 --2- 172.21.15.115:0/2978593494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87d81007a0 0x7f87d8101ec0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.402+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 >> 172.21.15.115:0/2978593494 conn(0x7f87d8075c90 msgr2=0x7f87d80689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.402+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 shutdown_connections 2023-07-14T18:10:46.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:46.402+0000 7f87de2d6700 1 -- 172.21.15.115:0/2978593494 wait complete. 2023-07-14T18:10:46.818 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:46 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2978593494' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T18:10:46.819 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":17,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2625350855},{"type":"v1","addr":"172.21.15.115:6801","nonce":2625350855}]},"active_addr":"172.21.15.115:6801/2625350855","active_change":"2023-07-14T18:04:28.808612+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.115:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":1997805564}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":3292541966}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":427221873}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":3496563251}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":1754876121}]}]} 2023-07-14T18:10:46.825 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-14T18:10:46.826 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-14T18:10:46.826 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T18:10:46.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:46 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2978593494' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T18:10:46.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:46 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2978593494' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T18:10:47.376 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:47.826 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:47 smithi115 ceph-mon[127451]: pgmap v266: 129 pgs: 129 active+clean; 623 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2023-07-14T18:10:47.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:47 smithi150 ceph-mon[128853]: pgmap v266: 129 pgs: 129 active+clean; 623 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2023-07-14T18:10:47.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:47 smithi187 ceph-mon[129317]: pgmap v266: 129 pgs: 129 active+clean; 623 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2023-07-14T18:10:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.816+0000 7feeeeaa2700 1 -- 172.21.15.115:0/1290853548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee80fee40 msgr2=0x7feee80ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:48.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.816+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/1290853548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee80fee40 0x7feee80ff210 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feee000b0d0 tx=0x7feee000b3e0 comp rx=0 tx=0).stop 2023-07-14T18:10:48.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.816+0000 7feeeeaa2700 1 -- 172.21.15.115:0/1290853548 shutdown_connections 2023-07-14T18:10:48.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.816+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/1290853548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feee8103ee0 0x7feee810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:48.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.816+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/1290853548 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee80fee40 0x7feee80ff210 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:48.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.816+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/1290853548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feee80ff7e0 0x7feee81037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:48.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.816+0000 7feeeeaa2700 1 -- 172.21.15.115:0/1290853548 >> 172.21.15.115:0/1290853548 conn(0x7feee8075ca0 msgr2=0x7feee80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:48.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.816+0000 7feeeeaa2700 1 -- 172.21.15.115:0/1290853548 shutdown_connections 2023-07-14T18:10:48.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.816+0000 7feeeeaa2700 1 -- 172.21.15.115:0/1290853548 wait complete. 2023-07-14T18:10:48.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.817+0000 7feeeeaa2700 1 Processor -- start 2023-07-14T18:10:48.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.817+0000 7feeeeaa2700 1 -- start start 2023-07-14T18:10:48.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeeeaa2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feee80fee40 0x7feee8196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:48.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeeeaa2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feee80ff7e0 0x7feee81972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:48.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeeeaa2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee8103ee0 0x7feee81a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:48.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeeeaa2700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee81a2370 con 0x7feee8103ee0 2023-07-14T18:10:48.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeeeaa2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee81a24b0 con 0x7feee80ff7e0 2023-07-14T18:10:48.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeeeaa2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee81a25f0 con 0x7feee80fee40 2023-07-14T18:10:48.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeed03f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee8103ee0 0x7feee81a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:48.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeed03f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee8103ee0 0x7feee81a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53160/0 (socket says 172.21.15.115:53160) 2023-07-14T18:10:48.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeed03f700 1 -- 172.21.15.115:0/521181164 learned_addr learned my addr 172.21.15.115:0/521181164 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:48.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeec83e700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feee80fee40 0x7feee8196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:48.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feee7fff700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feee80ff7e0 0x7feee81972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:48.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeed03f700 1 -- 172.21.15.115:0/521181164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feee80fee40 msgr2=0x7feee8196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:48.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeed03f700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feee80fee40 0x7feee8196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:48.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeed03f700 1 -- 172.21.15.115:0/521181164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feee80ff7e0 msgr2=0x7feee81972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:48.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeed03f700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feee80ff7e0 0x7feee81972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:48.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeed03f700 1 -- 172.21.15.115:0/521181164 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0009d00 con 0x7feee8103ee0 2023-07-14T18:10:48.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeed03f700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee8103ee0 0x7feee81a1e30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7feed000b410 tx=0x7feed000b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:48.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feee5ffb700 1 -- 172.21.15.115:0/521181164 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0011830 con 0x7feee8103ee0 2023-07-14T18:10:48.860 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feee5ffb700 1 -- 172.21.15.115:0/521181164 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7feed0011e20 con 0x7feee8103ee0 2023-07-14T18:10:48.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee81a2850 con 0x7feee8103ee0 2023-07-14T18:10:48.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.818+0000 7feee5ffb700 1 -- 172.21.15.115:0/521181164 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0010cd0 con 0x7feee8103ee0 2023-07-14T18:10:48.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.819+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee81a2cd0 con 0x7feee8103ee0 2023-07-14T18:10:48.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.820+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee804e340 con 0x7feee8103ee0 2023-07-14T18:10:48.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.821+0000 7feee5ffb700 1 -- 172.21.15.115:0/521181164 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7feed0011990 con 0x7feee8103ee0 2023-07-14T18:10:48.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.821+0000 7feee5ffb700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feed40a11e0 0x7feed40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:48.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.821+0000 7feeec83e700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feed40a11e0 0x7feed40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:48.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.821+0000 7feee5ffb700 1 -- 172.21.15.115:0/521181164 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7feed00cc4f0 con 0x7feee8103ee0 2023-07-14T18:10:48.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.822+0000 7feeec83e700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feed40a11e0 0x7feed40a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feee0009fd0 tx=0x7feee000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:48.863 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:48.826+0000 7feee5ffb700 1 -- 172.21.15.115:0/521181164 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7feed0095a00 con 0x7feee8103ee0 2023-07-14T18:10:49.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.014+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feee81a2f50 con 0x7feee8103ee0 2023-07-14T18:10:49.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.016+0000 7feee5ffb700 1 -- 172.21.15.115:0/521181164 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7feed0099020 con 0x7feee8103ee0 2023-07-14T18:10:49.054 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:49.054 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:49.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feed40a11e0 msgr2=0x7feed40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:49.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feed40a11e0 0x7feed40a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feee0009fd0 tx=0x7feee000b040 comp rx=0 tx=0).stop 2023-07-14T18:10:49.056 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee8103ee0 msgr2=0x7feee81a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee8103ee0 0x7feee81a1e30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7feed000b410 tx=0x7feed000b720 comp rx=0 tx=0).stop 2023-07-14T18:10:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 shutdown_connections 2023-07-14T18:10:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7feed40a11e0 0x7feed40a3690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feee8103ee0 0x7feee81a1e30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feee80fee40 0x7feee8196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 --2- 172.21.15.115:0/521181164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feee80ff7e0 0x7feee81972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 >> 172.21.15.115:0/521181164 conn(0x7feee8075ca0 msgr2=0x7feee80fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 shutdown_connections 2023-07-14T18:10:49.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:49.018+0000 7feeeeaa2700 1 -- 172.21.15.115:0/521181164 wait complete. 2023-07-14T18:10:49.628 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-14T18:10:49.629 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T18:10:50.179 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:50.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:49 smithi115 ceph-mon[127451]: pgmap v267: 129 pgs: 129 active+clean; 623 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2023-07-14T18:10:50.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:49 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/521181164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:50.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:49 smithi150 ceph-mon[128853]: pgmap v267: 129 pgs: 129 active+clean; 623 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2023-07-14T18:10:50.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:49 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/521181164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:50.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:49 smithi187 ceph-mon[129317]: pgmap v267: 129 pgs: 129 active+clean; 623 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2023-07-14T18:10:50.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:49 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/521181164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:51.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 -- 172.21.15.115:0/1906394300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fef80 msgr2=0x7fe9e8102f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:51.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/1906394300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fef80 0x7fe9e8102f50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009990 tx=0x7fe9dc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 -- 172.21.15.115:0/1906394300 shutdown_connections 2023-07-14T18:10:51.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/1906394300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e8103680 0x7fe9e810b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/1906394300 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9e80fe5e0 0x7fe9e80fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/1906394300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fef80 0x7fe9e8102f50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 -- 172.21.15.115:0/1906394300 >> 172.21.15.115:0/1906394300 conn(0x7fe9e8075280 msgr2=0x7fe9e8075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:51.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 -- 172.21.15.115:0/1906394300 shutdown_connections 2023-07-14T18:10:51.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 -- 172.21.15.115:0/1906394300 wait complete. 2023-07-14T18:10:51.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 Processor -- start 2023-07-14T18:10:51.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.433+0000 7fe9ee83f700 1 -- start start 2023-07-14T18:10:51.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9ee83f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fe5e0 0x7fe9e8196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:51.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9ee83f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e80fef80 0x7fe9e8196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:51.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9ee83f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9e8103680 0x7fe9e81a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:51.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9ee83f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e8197900 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9ee83f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e81a2070 con 0x7fe9e80fef80 2023-07-14T18:10:51.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9ee83f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e81a21e0 con 0x7fe9e8103680 2023-07-14T18:10:51.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fe5e0 0x7fe9e8196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:51.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fe5e0 0x7fe9e8196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53166/0 (socket says 172.21.15.115:53166) 2023-07-14T18:10:51.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e7fff700 1 -- 172.21.15.115:0/4280015617 learned_addr learned my addr 172.21.15.115:0/4280015617 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:51.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e77fe700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e80fef80 0x7fe9e8196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:51.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9ecddc700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9e8103680 0x7fe9e81a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:51.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e7fff700 1 -- 172.21.15.115:0/4280015617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9e8103680 msgr2=0x7fe9e81a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:51.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e7fff700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9e8103680 0x7fe9e81a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e7fff700 1 -- 172.21.15.115:0/4280015617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e80fef80 msgr2=0x7fe9e8196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:51.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e7fff700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e80fef80 0x7fe9e8196ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e7fff700 1 -- 172.21.15.115:0/4280015617 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc0095f0 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9ecddc700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9e8103680 0x7fe9e81a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:10:51.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e7fff700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fe5e0 0x7fe9e8196ab0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800d950 tx=0x7fe9d800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:51.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e57fa700 1 -- 172.21.15.115:0/4280015617 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8009b40 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e81a24c0 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e57fa700 1 -- 172.21.15.115:0/4280015617 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe9d8009ca0 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.434+0000 7fe9e57fa700 1 -- 172.21.15.115:0/4280015617 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d800f620 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.435+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e81a29e0 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.435+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e804e340 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.437+0000 7fe9e57fa700 1 -- 172.21.15.115:0/4280015617 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fe9d8010460 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.437+0000 7fe9e57fa700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe9d00a11e0 0x7fe9d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:51.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.438+0000 7fe9e77fe700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe9d00a11e0 0x7fe9d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:51.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.438+0000 7fe9e57fa700 1 -- 172.21.15.115:0/4280015617 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe9d80cd020 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.438+0000 7fe9e77fe700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe9d00a11e0 0x7fe9d00a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0036c0 tx=0x7fe9dc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:51.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.440+0000 7fe9e57fa700 1 -- 172.21.15.115:0/4280015617 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe9d8096420 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.628+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9e810bd30 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.630+0000 7fe9e57fa700 1 -- 172.21.15.115:0/4280015617 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7fe9d8020090 con 0x7fe9e80fe5e0 2023-07-14T18:10:51.668 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:10:51.668 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:10:51.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe9d00a11e0 msgr2=0x7fe9d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:51.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe9d00a11e0 0x7fe9d00a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0036c0 tx=0x7fe9dc005fb0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fe5e0 msgr2=0x7fe9e8196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:51.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fe5e0 0x7fe9e8196ab0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800d950 tx=0x7fe9d800dd10 comp rx=0 tx=0).stop 2023-07-14T18:10:51.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 shutdown_connections 2023-07-14T18:10:51.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe9d00a11e0 0x7fe9d00a3690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9e8103680 0x7fe9e81a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9e80fe5e0 0x7fe9e8196ab0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 --2- 172.21.15.115:0/4280015617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9e80fef80 0x7fe9e8196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:51.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 >> 172.21.15.115:0/4280015617 conn(0x7fe9e8075280 msgr2=0x7fe9e80fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:51.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 shutdown_connections 2023-07-14T18:10:51.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:51.632+0000 7fe9ee83f700 1 -- 172.21.15.115:0/4280015617 wait complete. 2023-07-14T18:10:52.128 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-14T18:10:52.128 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-14T18:10:52.129 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-14T18:10:52.129 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-14T18:10:52.130 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-14T18:10:52.130 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-14T18:10:52.131 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-14T18:10:52.131 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-14T18:10:52.132 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-07-14T18:10:52.132 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-07-14T18:10:52.133 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-07-14T18:10:52.133 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-07-14T18:10:52.305 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:51 smithi115 ceph-mon[127451]: pgmap v268: 129 pgs: 129 active+clean; 623 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.8 KiB/s wr, 11 op/s 2023-07-14T18:10:52.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:51 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4280015617' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:52.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:51 smithi150 ceph-mon[128853]: pgmap v268: 129 pgs: 129 active+clean; 623 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.8 KiB/s wr, 11 op/s 2023-07-14T18:10:52.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:51 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/4280015617' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:52.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:51 smithi187 ceph-mon[129317]: pgmap v268: 129 pgs: 129 active+clean; 623 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.8 KiB/s wr, 11 op/s 2023-07-14T18:10:52.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:51 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/4280015617' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:10:52.900 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:52.945 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:52.958 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:52.998 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:53.091 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:53.092 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:53.094 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:53.165 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:53.195 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:53.232 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:53.236 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:53.586 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:53 smithi115 ceph-mon[127451]: pgmap v269: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.8 KiB/s wr, 13 op/s 2023-07-14T18:10:53.645 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:10:53.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:53 smithi150 ceph-mon[128853]: pgmap v269: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.8 KiB/s wr, 13 op/s 2023-07-14T18:10:53.933 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:53 smithi187 ceph-mon[129317]: pgmap v269: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.8 KiB/s wr, 13 op/s 2023-07-14T18:10:56.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:55 smithi115 ceph-mon[127451]: pgmap v270: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-07-14T18:10:56.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:55 smithi150 ceph-mon[128853]: pgmap v270: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-07-14T18:10:56.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:55 smithi187 ceph-mon[129317]: pgmap v270: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-07-14T18:10:58.265 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:57 smithi115 ceph-mon[127451]: pgmap v271: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T18:10:58.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:57 smithi150 ceph-mon[128853]: pgmap v271: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T18:10:58.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:57 smithi187 ceph-mon[129317]: pgmap v271: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T18:10:59.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 -- 172.21.15.115:0/3527002926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda340fee40 msgr2=0x7fda340ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:59.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 --2- 172.21.15.115:0/3527002926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda340fee40 0x7fda340ff210 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fda280099c0 tx=0x7fda28009cd0 comp rx=0 tx=0).stop 2023-07-14T18:10:59.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 -- 172.21.15.115:0/3527002926 shutdown_connections 2023-07-14T18:10:59.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 --2- 172.21.15.115:0/3527002926 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda34103ee0 0x7fda3410ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:59.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 --2- 172.21.15.115:0/3527002926 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda340fee40 0x7fda340ff210 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:59.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 --2- 172.21.15.115:0/3527002926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda340ff7e0 0x7fda341037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:59.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 -- 172.21.15.115:0/3527002926 >> 172.21.15.115:0/3527002926 conn(0x7fda34075ca0 msgr2=0x7fda340760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:59.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 -- 172.21.15.115:0/3527002926 shutdown_connections 2023-07-14T18:10:59.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 -- 172.21.15.115:0/3527002926 wait complete. 2023-07-14T18:10:59.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 Processor -- start 2023-07-14T18:10:59.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.175+0000 7fda3afc9700 1 -- start start 2023-07-14T18:10:59.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda3afc9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda340fee40 0x7fda3419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:59.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda3afc9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda340ff7e0 0x7fda3419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:59.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda3afc9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda34103ee0 0x7fda34195ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:59.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda3afc9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda34196130 con 0x7fda340fee40 2023-07-14T18:10:59.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda3afc9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda341962a0 con 0x7fda34103ee0 2023-07-14T18:10:59.770 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda3afc9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda34196460 con 0x7fda340ff7e0 2023-07-14T18:10:59.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda38d65700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda340fee40 0x7fda3419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:59.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda38d65700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda340fee40 0x7fda3419ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53180/0 (socket says 172.21.15.115:53180) 2023-07-14T18:10:59.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda38d65700 1 -- 172.21.15.115:0/1074357829 learned_addr learned my addr 172.21.15.115:0/1074357829 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:10:59.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda33fff700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda340ff7e0 0x7fda3419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:59.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda39566700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda34103ee0 0x7fda34195ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:59.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda38d65700 1 -- 172.21.15.115:0/1074357829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda340ff7e0 msgr2=0x7fda3419bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:59.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda38d65700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda340ff7e0 0x7fda3419bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:59.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda38d65700 1 -- 172.21.15.115:0/1074357829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda34103ee0 msgr2=0x7fda34195ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:59.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda38d65700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda34103ee0 0x7fda34195ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:59.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda38d65700 1 -- 172.21.15.115:0/1074357829 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda280095f0 con 0x7fda340fee40 2023-07-14T18:10:59.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda39566700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda34103ee0 0x7fda34195ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:10:59.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda38d65700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda340fee40 0x7fda3419ba40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fda28009990 tx=0x7fda28003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:59.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda31ffb700 1 -- 172.21.15.115:0/1074357829 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2801d070 con 0x7fda340fee40 2023-07-14T18:10:59.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda31ffb700 1 -- 172.21.15.115:0/1074357829 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fda28004160 con 0x7fda340fee40 2023-07-14T18:10:59.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda341966e0 con 0x7fda340fee40 2023-07-14T18:10:59.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.176+0000 7fda31ffb700 1 -- 172.21.15.115:0/1074357829 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28022dd0 con 0x7fda340fee40 2023-07-14T18:10:59.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.177+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda341a2a30 con 0x7fda340fee40 2023-07-14T18:10:59.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.178+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fda3404e340 con 0x7fda340fee40 2023-07-14T18:10:59.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.179+0000 7fda31ffb700 1 -- 172.21.15.115:0/1074357829 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fda28022570 con 0x7fda340fee40 2023-07-14T18:10:59.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.179+0000 7fda31ffb700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda240a11e0 0x7fda240a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:59.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.179+0000 7fda33fff700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda240a11e0 0x7fda240a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:59.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.179+0000 7fda31ffb700 1 -- 172.21.15.115:0/1074357829 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fda280ce020 con 0x7fda340fee40 2023-07-14T18:10:59.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.179+0000 7fda31ffb700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] conn(0x7fda240abd50 0x7fda240ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:10:59.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.179+0000 7fda31ffb700 1 -- 172.21.15.115:0/1074357829 --> [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda240ae850 con 0x7fda240abd50 2023-07-14T18:10:59.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.179+0000 7fda31ffb700 1 -- 172.21.15.115:0/1074357829 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda280d06a0 con 0x7fda340fee40 2023-07-14T18:10:59.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.180+0000 7fda33fff700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda240a11e0 0x7fda240a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fda34077370 tx=0x7fda2000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:59.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.180+0000 7fda39566700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] conn(0x7fda240abd50 0x7fda240ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:10:59.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.180+0000 7fda39566700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] conn(0x7fda240abd50 0x7fda240ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:10:59.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.181+0000 7fda31ffb700 1 -- 172.21.15.115:0/1074357829 <== osd.5 v2:172.21.15.187:6808/3577781836 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fda240ae850 con 0x7fda240abd50 2023-07-14T18:10:59.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.196+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 --> [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fda34068ab0 con 0x7fda240abd50 2023-07-14T18:10:59.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda31ffb700 1 -- 172.21.15.115:0/1074357829 <== osd.5 v2:172.21.15.187:6808/3577781836 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fda34068ab0 con 0x7fda240abd50 2023-07-14T18:10:59.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 >> [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] conn(0x7fda240abd50 msgr2=0x7fda240ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:59.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda3afc9700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] conn(0x7fda240abd50 0x7fda240ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:10:59.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda240a11e0 msgr2=0x7fda240a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:59.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda3afc9700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda240a11e0 0x7fda240a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fda34077370 tx=0x7fda2000a380 comp rx=0 tx=0).stop 2023-07-14T18:10:59.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda340fee40 msgr2=0x7fda3419ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:10:59.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda3afc9700 1 --2- 172.21.15.115:0/1074357829 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda340fee40 0x7fda3419ba40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fda28009990 tx=0x7fda28003820 comp rx=0 tx=0).stop 2023-07-14T18:10:59.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda39566700 1 -- 172.21.15.115:0/1074357829 reap_dead start 2023-07-14T18:10:59.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 shutdown_connections 2023-07-14T18:10:59.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 >> 172.21.15.115:0/1074357829 conn(0x7fda34075ca0 msgr2=0x7fda34068160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:10:59.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.197+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 shutdown_connections 2023-07-14T18:10:59.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:10:58.198+0000 7fda3afc9700 1 -- 172.21.15.115:0/1074357829 wait complete. 2023-07-14T18:11:00.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:10:59 smithi115 ceph-mon[127451]: pgmap v272: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-07-14T18:11:00.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.096+0000 7f02c1f1c700 1 -- 172.21.15.115:0/769926635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02bc0fff30 msgr2=0x7f02bc100300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.136 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.096+0000 7f02c1f1c700 1 --2- 172.21.15.115:0/769926635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02bc0fff30 0x7f02bc100300 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0099c0 tx=0x7f02ac009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.096+0000 7f02c1f1c700 1 -- 172.21.15.115:0/769926635 shutdown_connections 2023-07-14T18:11:00.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.096+0000 7f02c1f1c700 1 --2- 172.21.15.115:0/769926635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02bc104fd0 0x7f02bc10cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.096+0000 7f02c1f1c700 1 --2- 172.21.15.115:0/769926635 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02bc0fff30 0x7f02bc100300 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.096+0000 7f02c1f1c700 1 --2- 172.21.15.115:0/769926635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02bc1008d0 0x7f02bc1048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.096+0000 7f02c1f1c700 1 -- 172.21.15.115:0/769926635 >> 172.21.15.115:0/769926635 conn(0x7f02bc0fb790 msgr2=0x7f02bc0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:00.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.096+0000 7f02c1f1c700 1 -- 172.21.15.115:0/769926635 shutdown_connections 2023-07-14T18:11:00.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.097+0000 7f02c1f1c700 1 -- 172.21.15.115:0/769926635 wait complete. 2023-07-14T18:11:00.137 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.097+0000 7f02c1f1c700 1 Processor -- start 2023-07-14T18:11:00.138 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.097+0000 7f02c1f1c700 1 -- start start 2023-07-14T18:11:00.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02c1f1c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02bc0fff30 0x7f02bc19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02c1f1c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02bc1008d0 0x7f02bc19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02c1f1c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02bc104fd0 0x7f02bc195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.139 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02c1f1c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc196110 con 0x7f02bc0fff30 2023-07-14T18:11:00.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02c1f1c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc196280 con 0x7f02bc104fd0 2023-07-14T18:11:00.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02c1f1c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc196440 con 0x7f02bc1008d0 2023-07-14T18:11:00.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bb7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02bc0fff30 0x7f02bc19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bb7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02bc0fff30 0x7f02bc19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53194/0 (socket says 172.21.15.115:53194) 2023-07-14T18:11:00.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bb7fe700 1 -- 172.21.15.115:0/2453272728 learned_addr learned my addr 172.21.15.115:0/2453272728 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:00.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02baffd700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02bc1008d0 0x7f02bc19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bbfff700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02bc104fd0 0x7f02bc195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.140 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bb7fe700 1 -- 172.21.15.115:0/2453272728 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02bc1008d0 msgr2=0x7f02bc19bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bb7fe700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f02bc1008d0 0x7f02bc19bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bb7fe700 1 -- 172.21.15.115:0/2453272728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02bc104fd0 msgr2=0x7f02bc195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bb7fe700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02bc104fd0 0x7f02bc195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bb7fe700 1 -- 172.21.15.115:0/2453272728 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac0095f0 con 0x7f02bc0fff30 2023-07-14T18:11:00.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bbfff700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02bc104fd0 0x7f02bc195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:00.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02bb7fe700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02bc0fff30 0x7f02bc19ba20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009f50 tx=0x7f02ac0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:00.141 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02b8ff9700 1 -- 172.21.15.115:0/2453272728 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac01d070 con 0x7f02bc0fff30 2023-07-14T18:11:00.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02b8ff9700 1 -- 172.21.15.115:0/2453272728 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f02ac0040b0 con 0x7f02bc0fff30 2023-07-14T18:11:00.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02bc1966c0 con 0x7f02bc0fff30 2023-07-14T18:11:00.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02b8ff9700 1 -- 172.21.15.115:0/2453272728 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac022db0 con 0x7f02bc0fff30 2023-07-14T18:11:00.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.098+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02bc1a2a10 con 0x7f02bc0fff30 2023-07-14T18:11:00.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.099+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02bc04e340 con 0x7f02bc0fff30 2023-07-14T18:11:00.142 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.100+0000 7f02b8ff9700 1 -- 172.21.15.115:0/2453272728 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f02ac022470 con 0x7f02bc0fff30 2023-07-14T18:11:00.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.101+0000 7f02b8ff9700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f02a40a1260 0x7f02a40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.101+0000 7f02baffd700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f02a40a1260 0x7f02a40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.101+0000 7f02b8ff9700 1 -- 172.21.15.115:0/2453272728 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f02ac0ce9d0 con 0x7f02bc0fff30 2023-07-14T18:11:00.143 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.101+0000 7f02b8ff9700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] conn(0x7f02a40abdd0 0x7f02a40ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.101+0000 7f02b8ff9700 1 -- 172.21.15.115:0/2453272728 --> [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02a40ae8d0 con 0x7f02a40abdd0 2023-07-14T18:11:00.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.101+0000 7f02b8ff9700 1 -- 172.21.15.115:0/2453272728 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02ac0cede0 con 0x7f02bc0fff30 2023-07-14T18:11:00.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.101+0000 7f02bbfff700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] conn(0x7f02a40abdd0 0x7f02a40ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.101+0000 7f02baffd700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f02a40a1260 0x7f02a40a3710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02bc106380 tx=0x7f02b000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:00.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.101+0000 7f02bbfff700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] conn(0x7f02a40abdd0 0x7f02a40ae200 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:00.144 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.102+0000 7f02b8ff9700 1 -- 172.21.15.115:0/2453272728 <== osd.6 v2:172.21.15.115:6818/1235073533 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f02a40ae8d0 con 0x7f02a40abdd0 2023-07-14T18:11:00.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.117+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 --> [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f02bc0fcb90 con 0x7f02a40abdd0 2023-07-14T18:11:00.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02b8ff9700 1 -- 172.21.15.115:0/2453272728 <== osd.6 v2:172.21.15.115:6818/1235073533 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f02bc0fcb90 con 0x7f02a40abdd0 2023-07-14T18:11:00.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] conn(0x7f02a40abdd0 msgr2=0x7f02a40ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] conn(0x7f02a40abdd0 0x7f02a40ae200 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f02a40a1260 msgr2=0x7f02a40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f02a40a1260 0x7f02a40a3710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02bc106380 tx=0x7f02b000a380 comp rx=0 tx=0).stop 2023-07-14T18:11:00.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02bc0fff30 msgr2=0x7f02bc19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 --2- 172.21.15.115:0/2453272728 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f02bc0fff30 0x7f02bc19ba20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009f50 tx=0x7f02ac0057b0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02bbfff700 1 -- 172.21.15.115:0/2453272728 reap_dead start 2023-07-14T18:11:00.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 shutdown_connections 2023-07-14T18:11:00.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 >> 172.21.15.115:0/2453272728 conn(0x7f02bc0fb790 msgr2=0x7f02bc0fc240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:00.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 shutdown_connections 2023-07-14T18:11:00.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.118+0000 7f02c1f1c700 1 -- 172.21.15.115:0/2453272728 wait complete. 2023-07-14T18:11:00.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:10:59 smithi150 ceph-mon[128853]: pgmap v272: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-07-14T18:11:00.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:10:59 smithi187 ceph-mon[129317]: pgmap v272: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-07-14T18:11:00.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.810+0000 7fecfa9b8700 1 -- 172.21.15.115:0/834429615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecf4104c70 msgr2=0x7fecf4105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.810+0000 7fecfa9b8700 1 --2- 172.21.15.115:0/834429615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecf4104c70 0x7fecf4105040 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fece4009990 tx=0x7fece4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.810+0000 7fecfa9b8700 1 -- 172.21.15.115:0/834429615 shutdown_connections 2023-07-14T18:11:00.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.810+0000 7fecfa9b8700 1 --2- 172.21.15.115:0/834429615 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecf40febc0 0x7fecf410cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.810+0000 7fecfa9b8700 1 --2- 172.21.15.115:0/834429615 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecf4104c70 0x7fecf4105040 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.810+0000 7fecfa9b8700 1 --2- 172.21.15.115:0/834429615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf40fe210 0x7fecf40fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.810+0000 7fecfa9b8700 1 -- 172.21.15.115:0/834429615 >> 172.21.15.115:0/834429615 conn(0x7fecf4075280 msgr2=0x7fecf4075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:00.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.810+0000 7fecfa9b8700 1 -- 172.21.15.115:0/834429615 shutdown_connections 2023-07-14T18:11:00.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.810+0000 7fecfa9b8700 1 -- 172.21.15.115:0/834429615 wait complete. 2023-07-14T18:11:00.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.811+0000 7fecfa9b8700 1 Processor -- start 2023-07-14T18:11:00.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.811+0000 7fecfa9b8700 1 -- start start 2023-07-14T18:11:00.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecfa9b8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecf40fe210 0x7fecf4195ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecfa9b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecf40febc0 0x7fecf41961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecfa9b8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf4104c70 0x7fecf41995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecfa9b8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf4199ba0 con 0x7fecf40fe210 2023-07-14T18:11:00.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecfa9b8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf4199d10 con 0x7fecf4104c70 2023-07-14T18:11:00.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecfa9b8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf4199ed0 con 0x7fecf40febc0 2023-07-14T18:11:00.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecf40fe210 0x7fecf4195ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecf40fe210 0x7fecf4195ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53210/0 (socket says 172.21.15.115:53210) 2023-07-14T18:11:00.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf3fff700 1 -- 172.21.15.115:0/429815090 learned_addr learned my addr 172.21.15.115:0/429815090 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:00.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf37fe700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecf40febc0 0x7fecf41961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf8f55700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf4104c70 0x7fecf41995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf3fff700 1 -- 172.21.15.115:0/429815090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecf40febc0 msgr2=0x7fecf41961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf3fff700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecf40febc0 0x7fecf41961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf3fff700 1 -- 172.21.15.115:0/429815090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf4104c70 msgr2=0x7fecf41995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf3fff700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecf4104c70 0x7fecf41995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf3fff700 1 -- 172.21.15.115:0/429815090 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece40095f0 con 0x7fecf40fe210 2023-07-14T18:11:00.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.812+0000 7fecf3fff700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecf40fe210 0x7fecf4195ca0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fece4005740 tx=0x7fece4005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:00.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.813+0000 7fecf17fa700 1 -- 172.21.15.115:0/429815090 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece401d070 con 0x7fecf40fe210 2023-07-14T18:11:00.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.813+0000 7fecf17fa700 1 -- 172.21.15.115:0/429815090 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fece4022470 con 0x7fecf40fe210 2023-07-14T18:11:00.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.813+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf419a150 con 0x7fecf40fe210 2023-07-14T18:11:00.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.813+0000 7fecf17fa700 1 -- 172.21.15.115:0/429815090 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4017650 con 0x7fecf40fe210 2023-07-14T18:11:00.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.813+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf419a420 con 0x7fecf40fe210 2023-07-14T18:11:00.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.814+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fecf410fe70 con 0x7fecf40fe210 2023-07-14T18:11:00.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.815+0000 7fecf17fa700 1 -- 172.21.15.115:0/429815090 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fece400fb00 con 0x7fecf40fe210 2023-07-14T18:11:00.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.816+0000 7fecf17fa700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecdc0a11e0 0x7fecdc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.816+0000 7fecf37fe700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecdc0a11e0 0x7fecdc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.816+0000 7fecf17fa700 1 -- 172.21.15.115:0/429815090 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fece40cec40 con 0x7fecf40fe210 2023-07-14T18:11:00.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.816+0000 7fecf17fa700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] conn(0x7fecdc0abd50 0x7fecdc0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:00.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.816+0000 7fecf17fa700 1 -- 172.21.15.115:0/429815090 --> [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecdc0ae850 con 0x7fecdc0abd50 2023-07-14T18:11:00.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.816+0000 7fecf17fa700 1 -- 172.21.15.115:0/429815090 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fece40cf050 con 0x7fecf40fe210 2023-07-14T18:11:00.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.816+0000 7fecf8f55700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] conn(0x7fecdc0abd50 0x7fecdc0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:00.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.816+0000 7fecf37fe700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecdc0a11e0 0x7fecdc0a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fecf4197280 tx=0x7fece800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:00.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.817+0000 7fecf8f55700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] conn(0x7fecdc0abd50 0x7fecdc0ae180 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:00.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.817+0000 7fecf17fa700 1 -- 172.21.15.115:0/429815090 <== osd.0 v2:172.21.15.115:6802/2677759926 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fecdc0ae850 con 0x7fecdc0abd50 2023-07-14T18:11:00.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.838+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 --> [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fecf404e340 con 0x7fecdc0abd50 2023-07-14T18:11:00.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.838+0000 7fecf17fa700 1 -- 172.21.15.115:0/429815090 <== osd.0 v2:172.21.15.115:6802/2677759926 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fecf404e340 con 0x7fecdc0abd50 2023-07-14T18:11:00.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.838+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] conn(0x7fecdc0abd50 msgr2=0x7fecdc0ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.838+0000 7fecfa9b8700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] conn(0x7fecdc0abd50 0x7fecdc0ae180 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:00.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.838+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecdc0a11e0 msgr2=0x7fecdc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.838+0000 7fecfa9b8700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fecdc0a11e0 0x7fecdc0a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fecf4197280 tx=0x7fece800a300 comp rx=0 tx=0).stop 2023-07-14T18:11:00.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.838+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecf40fe210 msgr2=0x7fecf4195ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:00.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.838+0000 7fecfa9b8700 1 --2- 172.21.15.115:0/429815090 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fecf40fe210 0x7fecf4195ca0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fece4005740 tx=0x7fece4005770 comp rx=0 tx=0).stop 2023-07-14T18:11:00.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.839+0000 7fecf8f55700 1 -- 172.21.15.115:0/429815090 reap_dead start 2023-07-14T18:11:00.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.839+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 shutdown_connections 2023-07-14T18:11:00.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.839+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 >> 172.21.15.115:0/429815090 conn(0x7fecf4075280 msgr2=0x7fecf40fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:00.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.839+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 shutdown_connections 2023-07-14T18:11:00.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:00.839+0000 7fecfa9b8700 1 -- 172.21.15.115:0/429815090 wait complete. 2023-07-14T18:11:01.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.308+0000 7f92eccfe700 1 -- 172.21.15.115:0/1171397129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92e80ff7e0 msgr2=0x7f92e81037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.347 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.308+0000 7f92eccfe700 1 --2- 172.21.15.115:0/1171397129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92e80ff7e0 0x7f92e81037b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f92d00099c0 tx=0x7f92d0009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.308+0000 7f92eccfe700 1 -- 172.21.15.115:0/1171397129 shutdown_connections 2023-07-14T18:11:01.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.308+0000 7f92eccfe700 1 --2- 172.21.15.115:0/1171397129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e8103ee0 0x7f92e810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.308+0000 7f92eccfe700 1 --2- 172.21.15.115:0/1171397129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e80fee40 0x7f92e80ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.308+0000 7f92eccfe700 1 --2- 172.21.15.115:0/1171397129 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92e80ff7e0 0x7f92e81037b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.308+0000 7f92eccfe700 1 -- 172.21.15.115:0/1171397129 >> 172.21.15.115:0/1171397129 conn(0x7f92e8075ca0 msgr2=0x7f92e80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:01.348 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.308+0000 7f92eccfe700 1 -- 172.21.15.115:0/1171397129 shutdown_connections 2023-07-14T18:11:01.349 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.308+0000 7f92eccfe700 1 -- 172.21.15.115:0/1171397129 wait complete. 2023-07-14T18:11:01.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92eccfe700 1 Processor -- start 2023-07-14T18:11:01.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92eccfe700 1 -- start start 2023-07-14T18:11:01.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92eccfe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92e80fee40 0x7f92e8196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.350 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92eccfe700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e80ff7e0 0x7f92e8197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92eccfe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e8103ee0 0x7f92e81a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92eccfe700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e8197990 con 0x7f92e80fee40 2023-07-14T18:11:01.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92eccfe700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e81a2100 con 0x7f92e8103ee0 2023-07-14T18:11:01.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92eccfe700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e81a2270 con 0x7f92e80ff7e0 2023-07-14T18:11:01.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92e659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92e80fee40 0x7f92e8196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92e659c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92e80fee40 0x7f92e8196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53242/0 (socket says 172.21.15.115:53242) 2023-07-14T18:11:01.351 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92e659c700 1 -- 172.21.15.115:0/688862463 learned_addr learned my addr 172.21.15.115:0/688862463 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:01.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.309+0000 7f92e5d9b700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e80ff7e0 0x7f92e8197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92e6d9d700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e8103ee0 0x7f92e81a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92e659c700 1 -- 172.21.15.115:0/688862463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e80ff7e0 msgr2=0x7f92e8197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92e659c700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e80ff7e0 0x7f92e8197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92e659c700 1 -- 172.21.15.115:0/688862463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e8103ee0 msgr2=0x7f92e81a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.352 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92e659c700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92e8103ee0 0x7f92e81a1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92e659c700 1 -- 172.21.15.115:0/688862463 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00095f0 con 0x7f92e80fee40 2023-07-14T18:11:01.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92e5d9b700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92e80ff7e0 0x7f92e8197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:01.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92e659c700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92e80fee40 0x7f92e8196b40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f92e0007f10 tx=0x7f92e000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:01.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92df7fe700 1 -- 172.21.15.115:0/688862463 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e000de90 con 0x7f92e80fee40 2023-07-14T18:11:01.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e81a2550 con 0x7f92e80fee40 2023-07-14T18:11:01.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92df7fe700 1 -- 172.21.15.115:0/688862463 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f92e000f040 con 0x7f92e80fee40 2023-07-14T18:11:01.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92df7fe700 1 -- 172.21.15.115:0/688862463 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e0015400 con 0x7f92e80fee40 2023-07-14T18:11:01.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.310+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e81a2a70 con 0x7f92e80fee40 2023-07-14T18:11:01.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.311+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f92e804e340 con 0x7f92e80fee40 2023-07-14T18:11:01.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.312+0000 7f92df7fe700 1 -- 172.21.15.115:0/688862463 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f92e0003a00 con 0x7f92e80fee40 2023-07-14T18:11:01.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.313+0000 7f92df7fe700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92d40a11e0 0x7f92d40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.313+0000 7f92e5d9b700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92d40a11e0 0x7f92d40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.313+0000 7f92df7fe700 1 -- 172.21.15.115:0/688862463 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f92e00cc8b0 con 0x7f92e80fee40 2023-07-14T18:11:01.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.313+0000 7f92df7fe700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] conn(0x7f92d40abbe0 0x7f92d40ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.313+0000 7f92df7fe700 1 -- 172.21.15.115:0/688862463 --> [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92d40ae6e0 con 0x7f92d40abbe0 2023-07-14T18:11:01.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.313+0000 7f92df7fe700 1 -- 172.21.15.115:0/688862463 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92e00d12c0 con 0x7f92e80fee40 2023-07-14T18:11:01.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.313+0000 7f92e5d9b700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92d40a11e0 0x7f92d40a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009990 tx=0x7f92d001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:01.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.313+0000 7f92e6d9d700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] conn(0x7f92d40abbe0 0x7f92d40ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.314+0000 7f92e6d9d700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] conn(0x7f92d40abbe0 0x7f92d40ae010 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:01.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.315+0000 7f92df7fe700 1 -- 172.21.15.115:0/688862463 <== osd.11 v2:172.21.15.187:6824/2714221245 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f92d40ae6e0 con 0x7f92d40abbe0 2023-07-14T18:11:01.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.330+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 --> [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f92e810ec30 con 0x7f92d40abbe0 2023-07-14T18:11:01.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92df7fe700 1 -- 172.21.15.115:0/688862463 <== osd.11 v2:172.21.15.187:6824/2714221245 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f92d8009800 con 0x7f92d40abbe0 2023-07-14T18:11:01.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 >> [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] conn(0x7f92d40abbe0 msgr2=0x7f92d40ae010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] conn(0x7f92d40abbe0 0x7f92d40ae010 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92d40a11e0 msgr2=0x7f92d40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f92d40a11e0 0x7f92d40a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009990 tx=0x7f92d001a040 comp rx=0 tx=0).stop 2023-07-14T18:11:01.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92e80fee40 msgr2=0x7f92e8196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 --2- 172.21.15.115:0/688862463 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92e80fee40 0x7f92e8196b40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f92e0007f10 tx=0x7f92e000d3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92e6d9d700 1 -- 172.21.15.115:0/688862463 reap_dead start 2023-07-14T18:11:01.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 shutdown_connections 2023-07-14T18:11:01.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 >> 172.21.15.115:0/688862463 conn(0x7f92e8075ca0 msgr2=0x7f92e8102790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:01.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 shutdown_connections 2023-07-14T18:11:01.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.331+0000 7f92eccfe700 1 -- 172.21.15.115:0/688862463 wait complete. 2023-07-14T18:11:01.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 -- 172.21.15.115:0/60986038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd194100f80 msgr2=0x7fd194101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 --2- 172.21.15.115:0/60986038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd194100f80 0x7fd194101350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd184009990 tx=0x7fd184009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 -- 172.21.15.115:0/60986038 shutdown_connections 2023-07-14T18:11:01.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 --2- 172.21.15.115:0/60986038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd19410a2d0 0x7fd19410c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 --2- 172.21.15.115:0/60986038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd194100f80 0x7fd194101350 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 --2- 172.21.15.115:0/60986038 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd194101890 0x7fd194109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 -- 172.21.15.115:0/60986038 >> 172.21.15.115:0/60986038 conn(0x7fd1940fa720 msgr2=0x7fd1940fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:01.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 -- 172.21.15.115:0/60986038 shutdown_connections 2023-07-14T18:11:01.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 -- 172.21.15.115:0/60986038 wait complete. 2023-07-14T18:11:01.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.214+0000 7fd1999f2700 1 Processor -- start 2023-07-14T18:11:01.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1999f2700 1 -- start start 2023-07-14T18:11:01.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1999f2700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd194100f80 0x7fd1940febe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1999f2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd194101890 0x7fd1940ff120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1999f2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd19410a2d0 0x7fd1940ff6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1999f2700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941a4020 con 0x7fd194101890 2023-07-14T18:11:01.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1999f2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941a4190 con 0x7fd19410a2d0 2023-07-14T18:11:01.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1999f2700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1941a4350 con 0x7fd194100f80 2023-07-14T18:11:01.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1927fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd194101890 0x7fd1940ff120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd192ffd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd194100f80 0x7fd1940febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1927fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd194101890 0x7fd1940ff120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53228/0 (socket says 172.21.15.115:53228) 2023-07-14T18:11:01.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1927fc700 1 -- 172.21.15.115:0/2441223373 learned_addr learned my addr 172.21.15.115:0/2441223373 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:01.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.215+0000 7fd1937fe700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd19410a2d0 0x7fd1940ff6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1927fc700 1 -- 172.21.15.115:0/2441223373 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd194100f80 msgr2=0x7fd1940febe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1927fc700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd194100f80 0x7fd1940febe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1927fc700 1 -- 172.21.15.115:0/2441223373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd19410a2d0 msgr2=0x7fd1940ff6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1927fc700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd19410a2d0 0x7fd1940ff6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1927fc700 1 -- 172.21.15.115:0/2441223373 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1840095f0 con 0x7fd194101890 2023-07-14T18:11:01.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd192ffd700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd194100f80 0x7fd1940febe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:11:01.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1927fc700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd194101890 0x7fd1940ff120 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c9b0 tx=0x7fd18800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:01.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1989f0700 1 -- 172.21.15.115:0/2441223373 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188007b00 con 0x7fd194101890 2023-07-14T18:11:01.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1989f0700 1 -- 172.21.15.115:0/2441223373 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd188007c60 con 0x7fd194101890 2023-07-14T18:11:01.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1941a4630 con 0x7fd194101890 2023-07-14T18:11:01.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1989f0700 1 -- 172.21.15.115:0/2441223373 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188018660 con 0x7fd194101890 2023-07-14T18:11:01.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.216+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1941a4b80 con 0x7fd194101890 2023-07-14T18:11:01.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.217+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd19410fd90 con 0x7fd194101890 2023-07-14T18:11:01.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.219+0000 7fd1989f0700 1 -- 172.21.15.115:0/2441223373 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fd18800f450 con 0x7fd194101890 2023-07-14T18:11:01.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.219+0000 7fd1989f0700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd17c0a1260 0x7fd17c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.219+0000 7fd192ffd700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd17c0a1260 0x7fd17c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.219+0000 7fd1989f0700 1 -- 172.21.15.115:0/2441223373 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd1880cd1c0 con 0x7fd194101890 2023-07-14T18:11:01.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.220+0000 7fd1989f0700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] conn(0x7fd17c0abdd0 0x7fd17c0ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:01.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.220+0000 7fd1989f0700 1 -- 172.21.15.115:0/2441223373 --> [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd17c0ae8d0 con 0x7fd17c0abdd0 2023-07-14T18:11:01.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.220+0000 7fd1989f0700 1 -- 172.21.15.115:0/2441223373 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1880cd660 con 0x7fd194101890 2023-07-14T18:11:01.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.220+0000 7fd192ffd700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd17c0a1260 0x7fd17c0a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd1840036c0 tx=0x7fd184005ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:01.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.220+0000 7fd1937fe700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] conn(0x7fd17c0abdd0 0x7fd17c0ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:01.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.220+0000 7fd1937fe700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] conn(0x7fd17c0abdd0 0x7fd17c0ae200 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:01.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.221+0000 7fd1989f0700 1 -- 172.21.15.115:0/2441223373 <== osd.1 v2:172.21.15.150:6800/396669416 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd17c0ae8d0 con 0x7fd17c0abdd0 2023-07-14T18:11:01.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.237+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 --> [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd19404e340 con 0x7fd17c0abdd0 2023-07-14T18:11:01.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.238+0000 7fd1989f0700 1 -- 172.21.15.115:0/2441223373 <== osd.1 v2:172.21.15.150:6800/396669416 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd19404e340 con 0x7fd17c0abdd0 2023-07-14T18:11:01.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.238+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 >> [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] conn(0x7fd17c0abdd0 msgr2=0x7fd17c0ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.238+0000 7fd1999f2700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] conn(0x7fd17c0abdd0 0x7fd17c0ae200 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.239+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd17c0a1260 msgr2=0x7fd17c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.239+0000 7fd1999f2700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd17c0a1260 0x7fd17c0a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd1840036c0 tx=0x7fd184005ab0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.239+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd194101890 msgr2=0x7fd1940ff120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:01.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.239+0000 7fd1999f2700 1 --2- 172.21.15.115:0/2441223373 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd194101890 0x7fd1940ff120 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c9b0 tx=0x7fd18800ccc0 comp rx=0 tx=0).stop 2023-07-14T18:11:01.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.239+0000 7fd1937fe700 1 -- 172.21.15.115:0/2441223373 reap_dead start 2023-07-14T18:11:01.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.239+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 shutdown_connections 2023-07-14T18:11:01.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.239+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 >> 172.21.15.115:0/2441223373 conn(0x7fd1940fa720 msgr2=0x7fd19410c480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:01.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.239+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 shutdown_connections 2023-07-14T18:11:01.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:01.239+0000 7fd1999f2700 1 -- 172.21.15.115:0/2441223373 wait complete. 2023-07-14T18:11:01.569 INFO:teuthology.orchestra.run.smithi115.stdout:158913789997 2023-07-14T18:11:01.569 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-14T18:11:02.266 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:01 smithi115 ceph-mon[127451]: pgmap v273: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2023-07-14T18:11:02.304 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:01 smithi187 ceph-mon[129317]: pgmap v273: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2023-07-14T18:11:02.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:01 smithi150 ceph-mon[128853]: pgmap v273: 129 pgs: 129 active+clean; 633 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2023-07-14T18:11:02.555 INFO:teuthology.orchestra.run.smithi115.stdout:335007449107 2023-07-14T18:11:02.555 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-07-14T18:11:02.657 INFO:teuthology.orchestra.run.smithi115.stdout:34359738433 2023-07-14T18:11:02.657 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-14T18:11:02.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 -- 172.21.15.115:0/1418370589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e80fee40 msgr2=0x7f50e80ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:02.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 --2- 172.21.15.115:0/1418370589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e80fee40 0x7f50e80ff210 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0099c0 tx=0x7f50dc009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:02.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 -- 172.21.15.115:0/1418370589 shutdown_connections 2023-07-14T18:11:02.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 --2- 172.21.15.115:0/1418370589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e8103ee0 0x7f50e810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:02.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 --2- 172.21.15.115:0/1418370589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e80fee40 0x7f50e80ff210 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:02.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 --2- 172.21.15.115:0/1418370589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e80ff7e0 0x7f50e81037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:02.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 -- 172.21.15.115:0/1418370589 >> 172.21.15.115:0/1418370589 conn(0x7f50e8075ca0 msgr2=0x7f50e80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:02.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 -- 172.21.15.115:0/1418370589 shutdown_connections 2023-07-14T18:11:02.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 -- 172.21.15.115:0/1418370589 wait complete. 2023-07-14T18:11:02.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 Processor -- start 2023-07-14T18:11:02.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.908+0000 7f50ee7aa700 1 -- start start 2023-07-14T18:11:02.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50ee7aa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e80fee40 0x7f50e819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:02.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50ee7aa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e80ff7e0 0x7f50e819e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:02.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50ee7aa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e8103ee0 0x7f50e8198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:02.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50ee7aa700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e81991b0 con 0x7f50e80ff7e0 2023-07-14T18:11:02.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50ee7aa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8199320 con 0x7f50e80fee40 2023-07-14T18:11:02.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50ee7aa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e81994e0 con 0x7f50e8103ee0 2023-07-14T18:11:02.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e80ff7e0 0x7f50e819e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:02.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e80ff7e0 0x7f50e819e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53258/0 (socket says 172.21.15.115:53258) 2023-07-14T18:11:02.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e77fe700 1 -- 172.21.15.115:0/3714931072 learned_addr learned my addr 172.21.15.115:0/3714931072 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:02.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e7fff700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e80fee40 0x7f50e819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:02.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50ecd47700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e8103ee0 0x7f50e8198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:02.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e77fe700 1 -- 172.21.15.115:0/3714931072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e8103ee0 msgr2=0x7f50e8198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:02.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e77fe700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e8103ee0 0x7f50e8198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:02.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e77fe700 1 -- 172.21.15.115:0/3714931072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e80fee40 msgr2=0x7f50e819de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:02.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e77fe700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50e80fee40 0x7f50e819de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:02.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e77fe700 1 -- 172.21.15.115:0/3714931072 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc0095f0 con 0x7f50e80ff7e0 2023-07-14T18:11:02.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50ecd47700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50e8103ee0 0x7f50e8198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.909+0000 7f50e77fe700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e80ff7e0 0x7f50e819e3a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f50d800d980 tx=0x7f50d800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.910+0000 7f50e57fa700 1 -- 172.21.15.115:0/3714931072 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d8009930 con 0x7f50e80ff7e0 2023-07-14T18:11:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.910+0000 7f50e57fa700 1 -- 172.21.15.115:0/3714931072 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f50d8010460 con 0x7f50e80ff7e0 2023-07-14T18:11:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.910+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e81997c0 con 0x7f50e80ff7e0 2023-07-14T18:11:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.910+0000 7f50e57fa700 1 -- 172.21.15.115:0/3714931072 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d800f620 con 0x7f50e80ff7e0 2023-07-14T18:11:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.910+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e81a4dd0 con 0x7f50e80ff7e0 2023-07-14T18:11:02.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.910+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50e810dae0 con 0x7f50e80ff7e0 2023-07-14T18:11:02.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.912+0000 7f50e57fa700 1 -- 172.21.15.115:0/3714931072 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f50d80105d0 con 0x7f50e80ff7e0 2023-07-14T18:11:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.912+0000 7f50e57fa700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f50d00a11e0 0x7f50d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.912+0000 7f50e7fff700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f50d00a11e0 0x7f50d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.912+0000 7f50e57fa700 1 -- 172.21.15.115:0/3714931072 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f50d80ccb20 con 0x7f50e80ff7e0 2023-07-14T18:11:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.912+0000 7f50e57fa700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] conn(0x7f50d00abcd0 0x7f50d00ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.912+0000 7f50e57fa700 1 -- 172.21.15.115:0/3714931072 --> [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50d00ae7d0 con 0x7f50d00abcd0 2023-07-14T18:11:02.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.912+0000 7f50e57fa700 1 -- 172.21.15.115:0/3714931072 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50d80ccf30 con 0x7f50e80ff7e0 2023-07-14T18:11:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.913+0000 7f50e7fff700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f50d00a11e0 0x7f50d00a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00b580 tx=0x7f50dc009f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.913+0000 7f50ecd47700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] conn(0x7f50d00abcd0 0x7f50d00ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.913+0000 7f50ecd47700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] conn(0x7f50d00abcd0 0x7f50d00ae100 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:02.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.914+0000 7f50e57fa700 1 -- 172.21.15.115:0/3714931072 <== osd.7 v2:172.21.15.150:6816/3128310736 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f50d00ae7d0 con 0x7f50d00abcd0 2023-07-14T18:11:02.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.930+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 --> [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50e804e340 con 0x7f50d00abcd0 2023-07-14T18:11:02.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50e57fa700 1 -- 172.21.15.115:0/3714931072 <== osd.7 v2:172.21.15.150:6816/3128310736 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f50e804e340 con 0x7f50d00abcd0 2023-07-14T18:11:02.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 >> [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] conn(0x7f50d00abcd0 msgr2=0x7f50d00ae100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:02.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] conn(0x7f50d00abcd0 0x7f50d00ae100 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:02.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f50d00a11e0 msgr2=0x7f50d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:02.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f50d00a11e0 0x7f50d00a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00b580 tx=0x7f50dc009f10 comp rx=0 tx=0).stop 2023-07-14T18:11:02.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e80ff7e0 msgr2=0x7f50e819e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:02.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 --2- 172.21.15.115:0/3714931072 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f50e80ff7e0 0x7f50e819e3a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f50d800d980 tx=0x7f50d800dc90 comp rx=0 tx=0).stop 2023-07-14T18:11:02.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ecd47700 1 -- 172.21.15.115:0/3714931072 reap_dead start 2023-07-14T18:11:02.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 shutdown_connections 2023-07-14T18:11:02.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 >> 172.21.15.115:0/3714931072 conn(0x7f50e8075ca0 msgr2=0x7f50e8102790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:02.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 shutdown_connections 2023-07-14T18:11:02.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:02.931+0000 7f50ee7aa700 1 -- 172.21.15.115:0/3714931072 wait complete. 2023-07-14T18:11:03.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.225+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/383961696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b94105000 msgr2=0x7f6b9410cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.225+0000 7f6b9c5e0700 1 --2- 172.21.15.115:0/383961696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b94105000 0x7f6b9410cb80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00b0d0 tx=0x7f6b8c00b3e0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.225+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/383961696 shutdown_connections 2023-07-14T18:11:03.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.225+0000 7f6b9c5e0700 1 --2- 172.21.15.115:0/383961696 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b94105000 0x7f6b9410cb80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.225+0000 7f6b9c5e0700 1 --2- 172.21.15.115:0/383961696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b940fffb0 0x7f6b94100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.225+0000 7f6b9c5e0700 1 --2- 172.21.15.115:0/383961696 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b94100950 0x7f6b941048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.225+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/383961696 >> 172.21.15.115:0/383961696 conn(0x7f6b94077cd0 msgr2=0x7f6b940780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:03.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.225+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/383961696 shutdown_connections 2023-07-14T18:11:03.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.225+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/383961696 wait complete. 2023-07-14T18:11:03.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b9c5e0700 1 Processor -- start 2023-07-14T18:11:03.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b9c5e0700 1 -- start start 2023-07-14T18:11:03.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b9c5e0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b940fffb0 0x7f6b94195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b9c5e0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b94100950 0x7f6b94196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b9c5e0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b94105000 0x7f6b9419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b9c5e0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9419ace0 con 0x7f6b94100950 2023-07-14T18:11:03.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b9c5e0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9419ae50 con 0x7f6b94105000 2023-07-14T18:11:03.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b9c5e0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9419b010 con 0x7f6b940fffb0 2023-07-14T18:11:03.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b99b7b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b94100950 0x7f6b94196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b99b7b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b94100950 0x7f6b94196260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53290/0 (socket says 172.21.15.115:53290) 2023-07-14T18:11:03.317 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.226+0000 7f6b99b7b700 1 -- 172.21.15.115:0/989084118 learned_addr learned my addr 172.21.15.115:0/989084118 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:03.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b9a37c700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b940fffb0 0x7f6b94195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b9ab7d700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b94105000 0x7f6b9419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b99b7b700 1 -- 172.21.15.115:0/989084118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b940fffb0 msgr2=0x7f6b94195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b99b7b700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b940fffb0 0x7f6b94195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b99b7b700 1 -- 172.21.15.115:0/989084118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b94105000 msgr2=0x7f6b9419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b99b7b700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6b94105000 0x7f6b9419a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b99b7b700 1 -- 172.21.15.115:0/989084118 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c009d00 con 0x7f6b94100950 2023-07-14T18:11:03.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b99b7b700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b94100950 0x7f6b94196260 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6b8400e970 tx=0x7f6b8400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:03.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b8b7fe700 1 -- 172.21.15.115:0/989084118 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b84009790 con 0x7f6b94100950 2023-07-14T18:11:03.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b8b7fe700 1 -- 172.21.15.115:0/989084118 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6b84004cc0 con 0x7f6b94100950 2023-07-14T18:11:03.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9419b2f0 con 0x7f6b94100950 2023-07-14T18:11:03.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b8b7fe700 1 -- 172.21.15.115:0/989084118 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b840055b0 con 0x7f6b94100950 2023-07-14T18:11:03.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.227+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b941a2b50 con 0x7f6b94100950 2023-07-14T18:11:03.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.228+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b9410fd00 con 0x7f6b94100950 2023-07-14T18:11:03.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.229+0000 7f6b8b7fe700 1 -- 172.21.15.115:0/989084118 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f6b84004e30 con 0x7f6b94100950 2023-07-14T18:11:03.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.230+0000 7f6b8b7fe700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b800a1260 0x7f6b800a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.230+0000 7f6b9a37c700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b800a1260 0x7f6b800a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.230+0000 7f6b8b7fe700 1 -- 172.21.15.115:0/989084118 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6b84014070 con 0x7f6b94100950 2023-07-14T18:11:03.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.230+0000 7f6b8b7fe700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] conn(0x7f6b800abdd0 0x7f6b800ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.230+0000 7f6b8b7fe700 1 -- 172.21.15.115:0/989084118 --> [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b800ae8d0 con 0x7f6b800abdd0 2023-07-14T18:11:03.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.230+0000 7f6b8b7fe700 1 -- 172.21.15.115:0/989084118 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b840cd500 con 0x7f6b94100950 2023-07-14T18:11:03.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.230+0000 7f6b9a37c700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b800a1260 0x7f6b800a3710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6b90005ea0 tx=0x7f6b90005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:03.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.230+0000 7f6b9ab7d700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] conn(0x7f6b800abdd0 0x7f6b800ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.231+0000 7f6b9ab7d700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] conn(0x7f6b800abdd0 0x7f6b800ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:03.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.231+0000 7f6b8b7fe700 1 -- 172.21.15.115:0/989084118 <== osd.4 v2:172.21.15.150:6808/3033896975 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6b800ae8d0 con 0x7f6b800abdd0 2023-07-14T18:11:03.321 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.247+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 --> [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6b9404e340 con 0x7f6b800abdd0 2023-07-14T18:11:03.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.247+0000 7f6b8b7fe700 1 -- 172.21.15.115:0/989084118 <== osd.4 v2:172.21.15.150:6808/3033896975 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6b9404e340 con 0x7f6b800abdd0 2023-07-14T18:11:03.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 >> [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] conn(0x7f6b800abdd0 msgr2=0x7f6b800ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] conn(0x7f6b800abdd0 0x7f6b800ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b800a1260 msgr2=0x7f6b800a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.322 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b800a1260 0x7f6b800a3710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6b90005ea0 tx=0x7f6b90005e30 comp rx=0 tx=0).stop 2023-07-14T18:11:03.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b94100950 msgr2=0x7f6b94196260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 --2- 172.21.15.115:0/989084118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b94100950 0x7f6b94196260 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6b8400e970 tx=0x7f6b8400ed30 comp rx=0 tx=0).stop 2023-07-14T18:11:03.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9ab7d700 1 -- 172.21.15.115:0/989084118 reap_dead start 2023-07-14T18:11:03.323 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 shutdown_connections 2023-07-14T18:11:03.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 >> 172.21.15.115:0/989084118 conn(0x7f6b94077cd0 msgr2=0x7f6b940fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:03.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 shutdown_connections 2023-07-14T18:11:03.324 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.248+0000 7f6b9c5e0700 1 -- 172.21.15.115:0/989084118 wait complete. 2023-07-14T18:11:03.341 INFO:teuthology.orchestra.run.smithi115.stdout:55834574908 2023-07-14T18:11:03.342 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-14T18:11:03.497 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:03 smithi115 ceph-mon[127451]: pgmap v274: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-07-14T18:11:03.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 -- 172.21.15.115:0/1591776203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f66b810a270 msgr2=0x7f66b810c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 --2- 172.21.15.115:0/1591776203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f66b810a270 0x7f66b810c6b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f66b000b0a0 tx=0x7f66b000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 -- 172.21.15.115:0/1591776203 shutdown_connections 2023-07-14T18:11:03.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 --2- 172.21.15.115:0/1591776203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f66b810a270 0x7f66b810c6b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 --2- 172.21.15.115:0/1591776203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b8100f20 0x7f66b81012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 --2- 172.21.15.115:0/1591776203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b8101830 0x7f66b8109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 -- 172.21.15.115:0/1591776203 >> 172.21.15.115:0/1591776203 conn(0x7f66b8075ca0 msgr2=0x7f66b80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:03.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 -- 172.21.15.115:0/1591776203 shutdown_connections 2023-07-14T18:11:03.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 -- 172.21.15.115:0/1591776203 wait complete. 2023-07-14T18:11:03.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 Processor -- start 2023-07-14T18:11:03.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.770+0000 7f66be267700 1 -- start start 2023-07-14T18:11:03.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66be267700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f66b8100f20 0x7f66b8195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66be267700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b8101830 0x7f66b81962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66be267700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b810a270 0x7f66b819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66be267700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b819ad50 con 0x7f66b8100f20 2023-07-14T18:11:03.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66be267700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b819aec0 con 0x7f66b810a270 2023-07-14T18:11:03.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66be267700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b819b080 con 0x7f66b8101830 2023-07-14T18:11:03.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f66b8100f20 0x7f66b8195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f66b8100f20 0x7f66b8195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53304/0 (socket says 172.21.15.115:53304) 2023-07-14T18:11:03.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b7fff700 1 -- 172.21.15.115:0/838168587 learned_addr learned my addr 172.21.15.115:0/838168587 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:03.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66bc804700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b810a270 0x7f66b819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b77fe700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b8101830 0x7f66b81962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b7fff700 1 -- 172.21.15.115:0/838168587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b8101830 msgr2=0x7f66b81962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b7fff700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66b8101830 0x7f66b81962d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b7fff700 1 -- 172.21.15.115:0/838168587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b810a270 msgr2=0x7f66b819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b7fff700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66b810a270 0x7f66b819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b7fff700 1 -- 172.21.15.115:0/838168587 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b0009d00 con 0x7f66b8100f20 2023-07-14T18:11:03.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.771+0000 7f66b7fff700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f66b8100f20 0x7f66b8195d90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f66a800e9a0 tx=0x7f66a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:03.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.772+0000 7f66b57fa700 1 -- 172.21.15.115:0/838168587 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8009790 con 0x7f66b8100f20 2023-07-14T18:11:03.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.772+0000 7f66b57fa700 1 -- 172.21.15.115:0/838168587 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f66a8010430 con 0x7f66b8100f20 2023-07-14T18:11:03.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.772+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b819b360 con 0x7f66b8100f20 2023-07-14T18:11:03.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.772+0000 7f66b57fa700 1 -- 172.21.15.115:0/838168587 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a800cea0 con 0x7f66b8100f20 2023-07-14T18:11:03.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.772+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b81a2bc0 con 0x7f66b8100f20 2023-07-14T18:11:03.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.773+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f66b804e340 con 0x7f66b8100f20 2023-07-14T18:11:03.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.774+0000 7f66b57fa700 1 -- 172.21.15.115:0/838168587 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f66a80105a0 con 0x7f66b8100f20 2023-07-14T18:11:03.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.774+0000 7f66b57fa700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f66a00a11e0 0x7f66a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.774+0000 7f66b77fe700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f66a00a11e0 0x7f66a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.775+0000 7f66b57fa700 1 -- 172.21.15.115:0/838168587 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f66a8014070 con 0x7f66b8100f20 2023-07-14T18:11:03.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.775+0000 7f66b57fa700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] conn(0x7f66a00abd50 0x7f66a00ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:03.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.775+0000 7f66b57fa700 1 -- 172.21.15.115:0/838168587 --> [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a00ae850 con 0x7f66a00abd50 2023-07-14T18:11:03.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.775+0000 7f66b57fa700 1 -- 172.21.15.115:0/838168587 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66a80cf3f0 con 0x7f66b8100f20 2023-07-14T18:11:03.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.775+0000 7f66b77fe700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f66a00a11e0 0x7f66a00a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f66b81973b0 tx=0x7f66ac009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:03.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.775+0000 7f66bc804700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] conn(0x7f66a00abd50 0x7f66a00ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:03.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.776+0000 7f66bc804700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] conn(0x7f66a00abd50 0x7f66a00ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:03.818 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.777+0000 7f66b57fa700 1 -- 172.21.15.115:0/838168587 <== osd.8 v2:172.21.15.187:6816/1513660091 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f66a00ae850 con 0x7f66a00abd50 2023-07-14T18:11:03.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.792+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 --> [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f66b810fcb0 con 0x7f66a00abd50 2023-07-14T18:11:03.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.792+0000 7f66b57fa700 1 -- 172.21.15.115:0/838168587 <== osd.8 v2:172.21.15.187:6816/1513660091 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f66b0009b80 con 0x7f66a00abd50 2023-07-14T18:11:03.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.792+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 >> [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] conn(0x7f66a00abd50 msgr2=0x7f66a00ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.792+0000 7f66be267700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] conn(0x7f66a00abd50 0x7f66a00ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.793+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f66a00a11e0 msgr2=0x7f66a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.793+0000 7f66be267700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f66a00a11e0 0x7f66a00a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f66b81973b0 tx=0x7f66ac009260 comp rx=0 tx=0).stop 2023-07-14T18:11:03.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.793+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f66b8100f20 msgr2=0x7f66b8195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:03.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.793+0000 7f66be267700 1 --2- 172.21.15.115:0/838168587 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f66b8100f20 0x7f66b8195d90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f66a800e9a0 tx=0x7f66a800ecb0 comp rx=0 tx=0).stop 2023-07-14T18:11:03.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.793+0000 7f66bc804700 1 -- 172.21.15.115:0/838168587 reap_dead start 2023-07-14T18:11:03.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.793+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 shutdown_connections 2023-07-14T18:11:03.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.793+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 >> 172.21.15.115:0/838168587 conn(0x7f66b8075ca0 msgr2=0x7f66b8101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:03.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.793+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 shutdown_connections 2023-07-14T18:11:03.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.793+0000 7f66be267700 1 -- 172.21.15.115:0/838168587 wait complete. 2023-07-14T18:11:03.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:03 smithi150 ceph-mon[128853]: pgmap v274: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-07-14T18:11:03.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:03 smithi187 ceph-mon[129317]: pgmap v274: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-07-14T18:11:04.333 INFO:teuthology.orchestra.run.smithi115.stdout:214748364837 2023-07-14T18:11:04.333 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-14T18:11:04.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.135+0000 7ff6670b5700 1 -- 172.21.15.115:0/1892898289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff660101150 msgr2=0x7ff66010bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:04.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.135+0000 7ff6670b5700 1 --2- 172.21.15.115:0/1892898289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff660101150 0x7ff66010bb40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff65800b0d0 tx=0x7ff65800b3e0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 -- 172.21.15.115:0/1892898289 shutdown_connections 2023-07-14T18:11:04.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 --2- 172.21.15.115:0/1892898289 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff660101150 0x7ff66010bb40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 --2- 172.21.15.115:0/1892898289 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6600fef40 0x7ff6600ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 --2- 172.21.15.115:0/1892898289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6601007a0 0x7ff660100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 -- 172.21.15.115:0/1892898289 >> 172.21.15.115:0/1892898289 conn(0x7ff660075c90 msgr2=0x7ff660076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:04.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 -- 172.21.15.115:0/1892898289 shutdown_connections 2023-07-14T18:11:04.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 -- 172.21.15.115:0/1892898289 wait complete. 2023-07-14T18:11:04.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 Processor -- start 2023-07-14T18:11:04.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 -- start start 2023-07-14T18:11:04.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6600fef40 0x7ff660198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6601007a0 0x7ff6601985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff660101150 0x7ff66019ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66019d020 con 0x7ff6601007a0 2023-07-14T18:11:04.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66019d190 con 0x7ff6600fef40 2023-07-14T18:11:04.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.136+0000 7ff6670b5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66019d350 con 0x7ff660101150 2023-07-14T18:11:04.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6601007a0 0x7ff6601985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6601007a0 0x7ff6601985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53274/0 (socket says 172.21.15.115:53274) 2023-07-14T18:11:04.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65ffff700 1 -- 172.21.15.115:0/2010363564 learned_addr learned my addr 172.21.15.115:0/2010363564 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:04.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff664e51700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6600fef40 0x7ff660198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff665652700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff660101150 0x7ff66019ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65ffff700 1 -- 172.21.15.115:0/2010363564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff660101150 msgr2=0x7ff66019ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:04.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65ffff700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff660101150 0x7ff66019ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65ffff700 1 -- 172.21.15.115:0/2010363564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6600fef40 msgr2=0x7ff660198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:04.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65ffff700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6600fef40 0x7ff660198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65ffff700 1 -- 172.21.15.115:0/2010363564 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff658009d00 con 0x7ff6601007a0 2023-07-14T18:11:04.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65ffff700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6601007a0 0x7ff6601985a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00e9a0 tx=0x7ff64c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:04.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65dffb700 1 -- 172.21.15.115:0/2010363564 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c009790 con 0x7ff6601007a0 2023-07-14T18:11:04.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66019d630 con 0x7ff6601007a0 2023-07-14T18:11:04.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65dffb700 1 -- 172.21.15.115:0/2010363564 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff64c010430 con 0x7ff6601007a0 2023-07-14T18:11:04.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.137+0000 7ff65dffb700 1 -- 172.21.15.115:0/2010363564 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c00cea0 con 0x7ff6601007a0 2023-07-14T18:11:04.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.138+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6601a4e90 con 0x7ff6601007a0 2023-07-14T18:11:04.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.138+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff66004e340 con 0x7ff6601007a0 2023-07-14T18:11:04.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.139+0000 7ff65dffb700 1 -- 172.21.15.115:0/2010363564 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff64c0105a0 con 0x7ff6601007a0 2023-07-14T18:11:04.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.140+0000 7ff65dffb700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff6500a11e0 0x7ff6500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.140+0000 7ff664e51700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff6500a11e0 0x7ff6500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.140+0000 7ff65dffb700 1 -- 172.21.15.115:0/2010363564 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff64c014070 con 0x7ff6601007a0 2023-07-14T18:11:04.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.140+0000 7ff65dffb700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] conn(0x7ff6500abd50 0x7ff6500ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.140+0000 7ff65dffb700 1 -- 172.21.15.115:0/2010363564 --> [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6500ae850 con 0x7ff6500abd50 2023-07-14T18:11:04.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.140+0000 7ff65dffb700 1 -- 172.21.15.115:0/2010363564 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff64c0cd990 con 0x7ff6601007a0 2023-07-14T18:11:04.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.140+0000 7ff665652700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] conn(0x7ff6500abd50 0x7ff6500ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.141+0000 7ff664e51700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff6500a11e0 0x7ff6500a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff654005950 tx=0x7ff65400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:04.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.141+0000 7ff665652700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] conn(0x7ff6500abd50 0x7ff6500ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:04.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.141+0000 7ff65dffb700 1 -- 172.21.15.115:0/2010363564 <== osd.9 v2:172.21.15.115:6826/3384303720 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff6500ae850 con 0x7ff6500abd50 2023-07-14T18:11:04.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.171+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 --> [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff66010ecf0 con 0x7ff6500abd50 2023-07-14T18:11:04.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff65dffb700 1 -- 172.21.15.115:0/2010363564 <== osd.9 v2:172.21.15.115:6826/3384303720 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff658009b80 con 0x7ff6500abd50 2023-07-14T18:11:04.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] conn(0x7ff6500abd50 msgr2=0x7ff6500ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:04.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] conn(0x7ff6500abd50 0x7ff6500ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff6500a11e0 msgr2=0x7ff6500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:04.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff6500a11e0 0x7ff6500a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff654005950 tx=0x7ff65400b410 comp rx=0 tx=0).stop 2023-07-14T18:11:04.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6601007a0 msgr2=0x7ff6601985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:04.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 --2- 172.21.15.115:0/2010363564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff6601007a0 0x7ff6601985a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00e9a0 tx=0x7ff64c00ecb0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff665652700 1 -- 172.21.15.115:0/2010363564 reap_dead start 2023-07-14T18:11:04.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 shutdown_connections 2023-07-14T18:11:04.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 >> 172.21.15.115:0/2010363564 conn(0x7ff660075c90 msgr2=0x7ff660068990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:04.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 shutdown_connections 2023-07-14T18:11:04.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:03.172+0000 7ff6670b5700 1 -- 172.21.15.115:0/2010363564 wait complete. 2023-07-14T18:11:04.662 INFO:teuthology.orchestra.run.smithi115.stdout:249108103201 2023-07-14T18:11:04.662 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-07-14T18:11:04.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fcb903c1700 1 -- 172.21.15.115:0/2883195691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb88103680 msgr2=0x7fcb8810b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:04.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fcb903c1700 1 --2- 172.21.15.115:0/2883195691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb88103680 0x7fcb8810b200 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400b0d0 tx=0x7fcb8400b3e0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fcb903c1700 1 -- 172.21.15.115:0/2883195691 shutdown_connections 2023-07-14T18:11:04.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fcb903c1700 1 --2- 172.21.15.115:0/2883195691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb88103680 0x7fcb8810b200 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fcb903c1700 1 --2- 172.21.15.115:0/2883195691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb880fe5e0 0x7fcb880fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fcb903c1700 1 --2- 172.21.15.115:0/2883195691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb880fef80 0x7fcb88102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fcb903c1700 1 -- 172.21.15.115:0/2883195691 >> 172.21.15.115:0/2883195691 conn(0x7fcb88075280 msgr2=0x7fcb88075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:04.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fcb903c1700 1 -- 172.21.15.115:0/2883195691 shutdown_connections 2023-07-14T18:11:04.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fcb903c1700 1 -- 172.21.15.115:0/2883195691 wait complete. 2023-07-14T18:11:04.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb903c1700 1 Processor -- start 2023-07-14T18:11:04.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb903c1700 1 -- start start 2023-07-14T18:11:04.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb903c1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb880fe5e0 0x7fcb88196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb903c1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb880fef80 0x7fcb88197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb903c1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb88103680 0x7fcb881a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb903c1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88197b40 con 0x7fcb880fef80 2023-07-14T18:11:04.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb903c1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb881a22b0 con 0x7fcb88103680 2023-07-14T18:11:04.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb903c1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb881a2420 con 0x7fcb880fe5e0 2023-07-14T18:11:04.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e15d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb880fe5e0 0x7fcb88196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e15d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb880fe5e0 0x7fcb88196cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.115:52482/0 (socket says 172.21.15.115:52482) 2023-07-14T18:11:04.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e15d700 1 -- 172.21.15.115:0/3683592951 learned_addr learned my addr 172.21.15.115:0/3683592951 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:04.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e95e700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb88103680 0x7fcb881a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.984 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8d95c700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb880fef80 0x7fcb88197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e15d700 1 -- 172.21.15.115:0/3683592951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb88103680 msgr2=0x7fcb881a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:04.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e15d700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb88103680 0x7fcb881a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e15d700 1 -- 172.21.15.115:0/3683592951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb880fef80 msgr2=0x7fcb88197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:04.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e15d700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb880fef80 0x7fcb88197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:04.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e15d700 1 -- 172.21.15.115:0/3683592951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb84009d00 con 0x7fcb880fe5e0 2023-07-14T18:11:04.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.941+0000 7fcb8e95e700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb88103680 0x7fcb881a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:04.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.942+0000 7fcb8d95c700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcb880fef80 0x7fcb88197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:04.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.942+0000 7fcb8e15d700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb880fe5e0 0x7fcb88196cf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000d950 tx=0x7fcb8000dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:04.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.942+0000 7fcb7f7fe700 1 -- 172.21.15.115:0/3683592951 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80009b40 con 0x7fcb880fe5e0 2023-07-14T18:11:04.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.942+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb881a2700 con 0x7fcb880fe5e0 2023-07-14T18:11:04.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.942+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb881a2c50 con 0x7fcb880fe5e0 2023-07-14T18:11:04.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.942+0000 7fcb7f7fe700 1 -- 172.21.15.115:0/3683592951 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcb80009ca0 con 0x7fcb880fe5e0 2023-07-14T18:11:04.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.942+0000 7fcb7f7fe700 1 -- 172.21.15.115:0/3683592951 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8000f620 con 0x7fcb880fe5e0 2023-07-14T18:11:04.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.943+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb6c000ff0 con 0x7fcb880fe5e0 2023-07-14T18:11:04.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.945+0000 7fcb7f7fe700 1 -- 172.21.15.115:0/3683592951 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fcb80010460 con 0x7fcb880fe5e0 2023-07-14T18:11:04.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.945+0000 7fcb7f7fe700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcb740a11e0 0x7fcb740a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.945+0000 7fcb8d95c700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcb740a11e0 0x7fcb740a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.946+0000 7fcb8d95c700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcb740a11e0 0x7fcb740a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb78005e20 tx=0x7fcb78005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:04.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.946+0000 7fcb7f7fe700 1 -- 172.21.15.115:0/3683592951 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcb8009a0a0 con 0x7fcb880fe5e0 2023-07-14T18:11:04.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.946+0000 7fcb7f7fe700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] conn(0x7fcb740abcd0 0x7fcb740ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:04.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.946+0000 7fcb7f7fe700 1 -- 172.21.15.115:0/3683592951 --> [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb740ae7d0 con 0x7fcb740abcd0 2023-07-14T18:11:04.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.946+0000 7fcb7f7fe700 1 -- 172.21.15.115:0/3683592951 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb80009e10 con 0x7fcb880fe5e0 2023-07-14T18:11:04.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.946+0000 7fcb8e95e700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] conn(0x7fcb740abcd0 0x7fcb740ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:04.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.947+0000 7fcb8e95e700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] conn(0x7fcb740abcd0 0x7fcb740ae100 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:04.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.947+0000 7fcb7f7fe700 1 -- 172.21.15.115:0/3683592951 <== osd.10 v2:172.21.15.150:6824/4251241728 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fcb740ae7d0 con 0x7fcb740abcd0 2023-07-14T18:11:05.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 -- 172.21.15.115:0/2555873976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe54c072980 msgr2=0x7fe54c072d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 --2- 172.21.15.115:0/2555873976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe54c072980 0x7fe54c072d50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009990 tx=0x7fe53c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 -- 172.21.15.115:0/2555873976 shutdown_connections 2023-07-14T18:11:05.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 --2- 172.21.15.115:0/2555873976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c10c570 0x7fe54c10e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 --2- 172.21.15.115:0/2555873976 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe54c072980 0x7fe54c072d50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 --2- 172.21.15.115:0/2555873976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe54c073290 0x7fe54c10c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.007 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 -- 172.21.15.115:0/2555873976 >> 172.21.15.115:0/2555873976 conn(0x7fe54c0fb790 msgr2=0x7fe54c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:05.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 -- 172.21.15.115:0/2555873976 shutdown_connections 2023-07-14T18:11:05.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 -- 172.21.15.115:0/2555873976 wait complete. 2023-07-14T18:11:05.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.916+0000 7fe552791700 1 Processor -- start 2023-07-14T18:11:05.008 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe552791700 1 -- start start 2023-07-14T18:11:05.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.966+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 --> [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcb6c002d70 con 0x7fcb740abcd0 2023-07-14T18:11:05.009 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe552791700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe54c072980 0x7fe54c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe552791700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c073290 0x7fe54c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe552791700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe54c10c570 0x7fe54c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe552791700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c19ce70 con 0x7fe54c10c570 2023-07-14T18:11:05.010 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe552791700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c19cfe0 con 0x7fe54c073290 2023-07-14T18:11:05.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe552791700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c19d1a0 con 0x7fe54c072980 2023-07-14T18:11:05.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe550d2e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe54c10c570 0x7fe54c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe550d2e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe54c10c570 0x7fe54c19c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53338/0 (socket says 172.21.15.115:53338) 2023-07-14T18:11:05.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe550d2e700 1 -- 172.21.15.115:0/1174471936 learned_addr learned my addr 172.21.15.115:0/1174471936 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:05.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe54bfff700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe54c072980 0x7fe54c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe54b7fe700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c073290 0x7fe54c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe550d2e700 1 -- 172.21.15.115:0/1174471936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe54c072980 msgr2=0x7fe54c197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe550d2e700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe54c072980 0x7fe54c197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe550d2e700 1 -- 172.21.15.115:0/1174471936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c073290 msgr2=0x7fe54c1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe550d2e700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c073290 0x7fe54c1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe550d2e700 1 -- 172.21.15.115:0/1174471936 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0095f0 con 0x7fe54c10c570 2023-07-14T18:11:05.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.917+0000 7fe550d2e700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe54c10c570 0x7fe54c19c860 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe5440094f0 tx=0x7fe54400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:05.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.918+0000 7fe5497fa700 1 -- 172.21.15.115:0/1174471936 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54400ddd0 con 0x7fe54c10c570 2023-07-14T18:11:05.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.918+0000 7fe5497fa700 1 -- 172.21.15.115:0/1174471936 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe54400f040 con 0x7fe54c10c570 2023-07-14T18:11:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.918+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c19d480 con 0x7fe54c10c570 2023-07-14T18:11:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.918+0000 7fe5497fa700 1 -- 172.21.15.115:0/1174471936 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544005a00 con 0x7fe54c10c570 2023-07-14T18:11:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.918+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c1a4ce0 con 0x7fe54c10c570 2023-07-14T18:11:05.014 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.919+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe54c111ec0 con 0x7fe54c10c570 2023-07-14T18:11:05.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.920+0000 7fe5497fa700 1 -- 172.21.15.115:0/1174471936 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fe544004f40 con 0x7fe54c10c570 2023-07-14T18:11:05.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.920+0000 7fe5497fa700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5340a11e0 0x7fe5340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.921+0000 7fe54bfff700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5340a11e0 0x7fe5340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.921+0000 7fe5497fa700 1 -- 172.21.15.115:0/1174471936 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe5440cd100 con 0x7fe54c10c570 2023-07-14T18:11:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.921+0000 7fe5497fa700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] conn(0x7fe5340abd50 0x7fe5340ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.921+0000 7fe5497fa700 1 -- 172.21.15.115:0/1174471936 --> [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5340ae850 con 0x7fe5340abd50 2023-07-14T18:11:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.921+0000 7fe5497fa700 1 -- 172.21.15.115:0/1174471936 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5440cd510 con 0x7fe54c10c570 2023-07-14T18:11:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.921+0000 7fe54bfff700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5340a11e0 0x7fe5340a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00b5c0 tx=0x7fe53c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.921+0000 7fe54b7fe700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] conn(0x7fe5340abd50 0x7fe5340ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.922+0000 7fe54b7fe700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] conn(0x7fe5340abd50 0x7fe5340ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.922+0000 7fe5497fa700 1 -- 172.21.15.115:0/1174471936 <== osd.2 v2:172.21.15.187:6800/262076123 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe5340ae850 con 0x7fe5340abd50 2023-07-14T18:11:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.938+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 --> [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe54c04e340 con 0x7fe5340abd50 2023-07-14T18:11:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.939+0000 7fe5497fa700 1 -- 172.21.15.115:0/1174471936 <== osd.2 v2:172.21.15.187:6800/262076123 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe54c04e340 con 0x7fe5340abd50 2023-07-14T18:11:05.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.939+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 >> [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] conn(0x7fe5340abd50 msgr2=0x7fe5340ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.939+0000 7fe552791700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] conn(0x7fe5340abd50 0x7fe5340ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.939+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5340a11e0 msgr2=0x7fe5340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.939+0000 7fe552791700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe5340a11e0 0x7fe5340a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00b5c0 tx=0x7fe53c005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.939+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe54c10c570 msgr2=0x7fe54c19c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.939+0000 7fe552791700 1 --2- 172.21.15.115:0/1174471936 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe54c10c570 0x7fe54c19c860 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe5440094f0 tx=0x7fe54400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.939+0000 7fe550d2e700 1 -- 172.21.15.115:0/1174471936 reap_dead start 2023-07-14T18:11:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 shutdown_connections 2023-07-14T18:11:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 >> 172.21.15.115:0/1174471936 conn(0x7fe54c0fb790 msgr2=0x7fe54c10e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:05.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 shutdown_connections 2023-07-14T18:11:05.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.940+0000 7fe552791700 1 -- 172.21.15.115:0/1174471936 wait complete. 2023-07-14T18:11:05.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.967+0000 7fcb7f7fe700 1 -- 172.21.15.115:0/3683592951 <== osd.10 v2:172.21.15.150:6824/4251241728 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcb6c002d70 con 0x7fcb740abcd0 2023-07-14T18:11:05.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.967+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 >> [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] conn(0x7fcb740abcd0 msgr2=0x7fcb740ae100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.967+0000 7fcb903c1700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] conn(0x7fcb740abcd0 0x7fcb740ae100 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.967+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcb740a11e0 msgr2=0x7fcb740a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.967+0000 7fcb903c1700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcb740a11e0 0x7fcb740a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcb78005e20 tx=0x7fcb78005db0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.967+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb880fe5e0 msgr2=0x7fcb88196cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.967+0000 7fcb903c1700 1 --2- 172.21.15.115:0/3683592951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcb880fe5e0 0x7fcb88196cf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000d950 tx=0x7fcb8000dd10 comp rx=0 tx=0).stop 2023-07-14T18:11:05.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.967+0000 7fcb8e95e700 1 -- 172.21.15.115:0/3683592951 reap_dead start 2023-07-14T18:11:05.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.968+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 shutdown_connections 2023-07-14T18:11:05.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.968+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 >> 172.21.15.115:0/3683592951 conn(0x7fcb88075280 msgr2=0x7fcb880fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:05.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.968+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 shutdown_connections 2023-07-14T18:11:05.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.968+0000 7fcb903c1700 1 -- 172.21.15.115:0/3683592951 wait complete. 2023-07-14T18:11:05.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.849+0000 7f6bae1be700 1 -- 172.21.15.115:0/918623279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba80fe5e0 msgr2=0x7f6ba80fe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.849+0000 7f6bae1be700 1 --2- 172.21.15.115:0/918623279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba80fe5e0 0x7f6ba80fe9b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009990 tx=0x7f6b98009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.849+0000 7f6bae1be700 1 -- 172.21.15.115:0/918623279 shutdown_connections 2023-07-14T18:11:05.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.849+0000 7f6bae1be700 1 --2- 172.21.15.115:0/918623279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ba8103680 0x7f6ba810b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.849+0000 7f6bae1be700 1 --2- 172.21.15.115:0/918623279 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba80fe5e0 0x7f6ba80fe9b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.849+0000 7f6bae1be700 1 --2- 172.21.15.115:0/918623279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ba80fef80 0x7f6ba8102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.849+0000 7f6bae1be700 1 -- 172.21.15.115:0/918623279 >> 172.21.15.115:0/918623279 conn(0x7f6ba8075280 msgr2=0x7f6ba8075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:05.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.849+0000 7f6bae1be700 1 -- 172.21.15.115:0/918623279 shutdown_connections 2023-07-14T18:11:05.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.850+0000 7f6bae1be700 1 -- 172.21.15.115:0/918623279 wait complete. 2023-07-14T18:11:05.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.850+0000 7f6bae1be700 1 Processor -- start 2023-07-14T18:11:05.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6bae1be700 1 -- start start 2023-07-14T18:11:05.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6bae1be700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ba80fe5e0 0x7f6ba819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6bae1be700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba80fef80 0x7f6ba819aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6bae1be700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ba8103680 0x7f6ba81949e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6bae1be700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba8195050 con 0x7f6ba80fef80 2023-07-14T18:11:05.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6bae1be700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba81951c0 con 0x7f6ba80fe5e0 2023-07-14T18:11:05.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6bae1be700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba8195380 con 0x7f6ba8103680 2023-07-14T18:11:05.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba80fef80 0x7f6ba819aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba80fef80 0x7f6ba819aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53322/0 (socket says 172.21.15.115:53322) 2023-07-14T18:11:05.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1989914807 learned_addr learned my addr 172.21.15.115:0/1989914807 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:05.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba77fe700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ba80fe5e0 0x7f6ba819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba7fff700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ba8103680 0x7f6ba81949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1989914807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ba8103680 msgr2=0x7f6ba81949e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba6ffd700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ba8103680 0x7f6ba81949e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1989914807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ba80fe5e0 msgr2=0x7f6ba819a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba6ffd700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ba80fe5e0 0x7f6ba819a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.851+0000 7f6ba6ffd700 1 -- 172.21.15.115:0/1989914807 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b980095f0 con 0x7f6ba80fef80 2023-07-14T18:11:05.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.852+0000 7f6ba6ffd700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba80fef80 0x7f6ba819aea0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00c9e0 tx=0x7f6b9c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:05.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.852+0000 7f6ba4ff9700 1 -- 172.21.15.115:0/1989914807 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c007aa0 con 0x7f6ba80fef80 2023-07-14T18:11:05.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.852+0000 7f6ba4ff9700 1 -- 172.21.15.115:0/1989914807 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6b9c007c00 con 0x7f6ba80fef80 2023-07-14T18:11:05.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.852+0000 7f6ba4ff9700 1 -- 172.21.15.115:0/1989914807 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c018660 con 0x7f6ba80fef80 2023-07-14T18:11:05.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.852+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba8195660 con 0x7f6ba80fef80 2023-07-14T18:11:05.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.852+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba810b910 con 0x7f6ba80fef80 2023-07-14T18:11:05.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.853+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ba810ebf0 con 0x7f6ba80fef80 2023-07-14T18:11:05.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.855+0000 7f6ba4ff9700 1 -- 172.21.15.115:0/1989914807 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f6b9c00f450 con 0x7f6ba80fef80 2023-07-14T18:11:05.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.855+0000 7f6ba4ff9700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b900a1260 0x7f6b900a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.856+0000 7f6ba77fe700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b900a1260 0x7f6b900a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.856+0000 7f6ba4ff9700 1 -- 172.21.15.115:0/1989914807 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6b9c0ccfb0 con 0x7f6ba80fef80 2023-07-14T18:11:05.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.856+0000 7f6ba4ff9700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] conn(0x7f6b900abdd0 0x7f6b900ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:05.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.856+0000 7f6ba4ff9700 1 -- 172.21.15.115:0/1989914807 --> [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b900ae8d0 con 0x7f6b900abdd0 2023-07-14T18:11:05.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.856+0000 7f6ba4ff9700 1 -- 172.21.15.115:0/1989914807 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b9c0cd3c0 con 0x7f6ba80fef80 2023-07-14T18:11:05.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.856+0000 7f6ba7fff700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] conn(0x7f6b900abdd0 0x7f6b900ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:05.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.856+0000 7f6ba77fe700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b900a1260 0x7f6b900a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800b5c0 tx=0x7f6b98005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:05.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.856+0000 7f6ba7fff700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] conn(0x7f6b900abdd0 0x7f6b900ae200 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:05.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.857+0000 7f6ba4ff9700 1 -- 172.21.15.115:0/1989914807 <== osd.3 v2:172.21.15.115:6810/3432098119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6b900ae8d0 con 0x7f6b900abdd0 2023-07-14T18:11:05.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.875+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 --> [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ba80fc7f0 con 0x7f6b900abdd0 2023-07-14T18:11:05.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.876+0000 7f6ba4ff9700 1 -- 172.21.15.115:0/1989914807 <== osd.3 v2:172.21.15.115:6810/3432098119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6ba80fc7f0 con 0x7f6b900abdd0 2023-07-14T18:11:05.380 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.876+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] conn(0x7f6b900abdd0 msgr2=0x7f6b900ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.876+0000 7f6bae1be700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] conn(0x7f6b900abdd0 0x7f6b900ae200 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.877+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b900a1260 msgr2=0x7f6b900a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.877+0000 7f6bae1be700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6b900a1260 0x7f6b900a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800b5c0 tx=0x7f6b98005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.877+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba80fef80 msgr2=0x7f6ba819aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:05.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.877+0000 7f6bae1be700 1 --2- 172.21.15.115:0/1989914807 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6ba80fef80 0x7f6ba819aea0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00c9e0 tx=0x7f6b9c00ccf0 comp rx=0 tx=0).stop 2023-07-14T18:11:05.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.877+0000 7f6ba7fff700 1 -- 172.21.15.115:0/1989914807 reap_dead start 2023-07-14T18:11:05.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.877+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 shutdown_connections 2023-07-14T18:11:05.381 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.877+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 >> 172.21.15.115:0/1989914807 conn(0x7f6ba8075280 msgr2=0x7f6ba80fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:05.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.877+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 shutdown_connections 2023-07-14T18:11:05.382 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:04.877+0000 7f6bae1be700 1 -- 172.21.15.115:0/1989914807 wait complete. 2023-07-14T18:11:05.382 INFO:teuthology.orchestra.run.smithi115.stdout:137438953522 2023-07-14T18:11:05.383 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-14T18:11:05.497 INFO:teuthology.orchestra.run.smithi115.stdout:193273528360 2023-07-14T18:11:05.497 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-14T18:11:05.784 INFO:teuthology.orchestra.run.smithi115.stdout:77309411385 2023-07-14T18:11:05.784 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-14T18:11:06.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:05 smithi115 ceph-mon[127451]: pgmap v275: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2023-07-14T18:11:06.262 INFO:teuthology.orchestra.run.smithi115.stdout:283467841565 2023-07-14T18:11:06.263 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-07-14T18:11:06.290 INFO:teuthology.orchestra.run.smithi115.stdout:304942678040 2023-07-14T18:11:06.290 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-07-14T18:11:06.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:05 smithi150 ceph-mon[128853]: pgmap v275: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2023-07-14T18:11:06.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:05 smithi187 ceph-mon[129317]: pgmap v275: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2023-07-14T18:11:06.603 INFO:teuthology.orchestra.run.smithi115.stdout:107374182453 2023-07-14T18:11:06.603 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-14T18:11:06.718 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:06.864 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:06.865 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:06.865 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:06.903 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:06.952 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:06.965 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:07.268 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:07.268 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:08.125 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:07 smithi115 ceph-mon[127451]: pgmap v276: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T18:11:08.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:07 smithi150 ceph-mon[128853]: pgmap v276: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T18:11:08.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:07 smithi187 ceph-mon[129317]: pgmap v276: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T18:11:10.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:09 smithi115 ceph-mon[127451]: pgmap v277: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T18:11:10.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:09 smithi150 ceph-mon[128853]: pgmap v277: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T18:11:10.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:09 smithi187 ceph-mon[129317]: pgmap v277: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T18:11:10.710 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:11.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.557+0000 7f5150b1c700 1 -- 172.21.15.115:0/3221480918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 msgr2=0x7f514c1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.557+0000 7f5150b1c700 1 --2- 172.21.15.115:0/3221480918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 0x7f514c1037b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f51400099c0 tx=0x7f5140009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.557+0000 7f5150b1c700 1 -- 172.21.15.115:0/3221480918 shutdown_connections 2023-07-14T18:11:11.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.557+0000 7f5150b1c700 1 --2- 172.21.15.115:0/3221480918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f514c103ee0 0x7f514c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.557+0000 7f5150b1c700 1 --2- 172.21.15.115:0/3221480918 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c0fee40 0x7f514c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.557+0000 7f5150b1c700 1 --2- 172.21.15.115:0/3221480918 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 0x7f514c1037b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.557+0000 7f5150b1c700 1 -- 172.21.15.115:0/3221480918 >> 172.21.15.115:0/3221480918 conn(0x7f514c075ca0 msgr2=0x7f514c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:11.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.557+0000 7f5150b1c700 1 -- 172.21.15.115:0/3221480918 shutdown_connections 2023-07-14T18:11:11.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.557+0000 7f5150b1c700 1 -- 172.21.15.115:0/3221480918 wait complete. 2023-07-14T18:11:11.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5150b1c700 1 Processor -- start 2023-07-14T18:11:11.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5150b1c700 1 -- start start 2023-07-14T18:11:11.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5150b1c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f514c0fee40 0x7f514c196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:11.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5150b1c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 0x7f514c196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:11.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5150b1c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c103ee0 0x7f514c1a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:11.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5150b1c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c197900 con 0x7f514c0ff7e0 2023-07-14T18:11:11.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5150b1c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c1a2070 con 0x7f514c0fee40 2023-07-14T18:11:11.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5150b1c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c1a21e0 con 0x7f514c103ee0 2023-07-14T18:11:11.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5149d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 0x7f514c196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:11.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5149d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 0x7f514c196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53368/0 (socket says 172.21.15.115:53368) 2023-07-14T18:11:11.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.558+0000 7f5149d9b700 1 -- 172.21.15.115:0/1457889871 learned_addr learned my addr 172.21.15.115:0/1457889871 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:11.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f514a59c700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f514c0fee40 0x7f514c196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:11.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f514ad9d700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c103ee0 0x7f514c1a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:11.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f5149d9b700 1 -- 172.21.15.115:0/1457889871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c103ee0 msgr2=0x7f514c1a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f5149d9b700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c103ee0 0x7f514c1a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f5149d9b700 1 -- 172.21.15.115:0/1457889871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f514c0fee40 msgr2=0x7f514c196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f5149d9b700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f514c0fee40 0x7f514c196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f5149d9b700 1 -- 172.21.15.115:0/1457889871 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51400095f0 con 0x7f514c0ff7e0 2023-07-14T18:11:11.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f514ad9d700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c103ee0 0x7f514c1a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:11.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f5149d9b700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 0x7f514c196ff0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f514000b580 tx=0x7f5140009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:11.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f513b7fe700 1 -- 172.21.15.115:0/1457889871 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514001d070 con 0x7f514c0ff7e0 2023-07-14T18:11:11.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c1a2460 con 0x7f514c0ff7e0 2023-07-14T18:11:11.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f513b7fe700 1 -- 172.21.15.115:0/1457889871 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5140003f40 con 0x7f514c0ff7e0 2023-07-14T18:11:11.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f513b7fe700 1 -- 172.21.15.115:0/1457889871 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5140022af0 con 0x7f514c0ff7e0 2023-07-14T18:11:11.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.559+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c1a2900 con 0x7f514c0ff7e0 2023-07-14T18:11:11.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.560+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c04e340 con 0x7f514c0ff7e0 2023-07-14T18:11:11.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.561+0000 7f513b7fe700 1 -- 172.21.15.115:0/1457889871 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f51400040b0 con 0x7f514c0ff7e0 2023-07-14T18:11:11.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.562+0000 7f513b7fe700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51340a11e0 0x7f51340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:11.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.562+0000 7f514a59c700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51340a11e0 0x7f51340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:11.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.562+0000 7f513b7fe700 1 -- 172.21.15.115:0/1457889871 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f51400ce340 con 0x7f514c0ff7e0 2023-07-14T18:11:11.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.562+0000 7f514a59c700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51340a11e0 0x7f51340a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f513c00ac50 tx=0x7f513c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:11.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.569+0000 7f513b7fe700 1 -- 172.21.15.115:0/1457889871 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f51400977d0 con 0x7f514c0ff7e0 2023-07-14T18:11:11.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.603+0000 7f2f0037a700 1 -- 172.21.15.115:0/4147994110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 msgr2=0x7f2ef810b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.603+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4147994110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 0x7f2ef810b200 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000b0d0 tx=0x7f2ef000b3e0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.603+0000 7f2f0037a700 1 -- 172.21.15.115:0/4147994110 shutdown_connections 2023-07-14T18:11:11.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.603+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4147994110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 0x7f2ef810b200 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.603+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4147994110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ef80fe5e0 0x7f2ef80fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.603+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4147994110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef80fef80 0x7f2ef8102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.603+0000 7f2f0037a700 1 -- 172.21.15.115:0/4147994110 >> 172.21.15.115:0/4147994110 conn(0x7f2ef8075280 msgr2=0x7f2ef8075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:11.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.603+0000 7f2f0037a700 1 -- 172.21.15.115:0/4147994110 shutdown_connections 2023-07-14T18:11:11.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.603+0000 7f2f0037a700 1 -- 172.21.15.115:0/4147994110 wait complete. 2023-07-14T18:11:11.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.604+0000 7f2f0037a700 1 Processor -- start 2023-07-14T18:11:11.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.604+0000 7f2f0037a700 1 -- start start 2023-07-14T18:11:11.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.604+0000 7f2f0037a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef80fe5e0 0x7f2ef8199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:11.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.604+0000 7f2f0037a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ef80fef80 0x7f2ef8199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:11.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.604+0000 7f2f0037a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 0x7f2ef81939f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:11.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.604+0000 7f2f0037a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef8194060 con 0x7f2ef8103680 2023-07-14T18:11:11.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.604+0000 7f2f0037a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef81941d0 con 0x7f2ef80fe5e0 2023-07-14T18:11:11.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.604+0000 7f2f0037a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef8194390 con 0x7f2ef80fef80 2023-07-14T18:11:11.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.604+0000 7f2efe917700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 0x7f2ef81939f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:11.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efe917700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 0x7f2ef81939f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53384/0 (socket says 172.21.15.115:53384) 2023-07-14T18:11:11.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efe917700 1 -- 172.21.15.115:0/4138692362 learned_addr learned my addr 172.21.15.115:0/4138692362 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:11.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efe116700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef80fe5e0 0x7f2ef8199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:11.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efd915700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ef80fef80 0x7f2ef8199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:11.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efe917700 1 -- 172.21.15.115:0/4138692362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ef80fef80 msgr2=0x7f2ef8199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efe917700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ef80fef80 0x7f2ef8199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efe917700 1 -- 172.21.15.115:0/4138692362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef80fe5e0 msgr2=0x7f2ef8199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efe917700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef80fe5e0 0x7f2ef8199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efe917700 1 -- 172.21.15.115:0/4138692362 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef0009d00 con 0x7f2ef8103680 2023-07-14T18:11:11.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2efe917700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 0x7f2ef81939f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0009380 tx=0x7f2ef00093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:11.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2eef7fe700 1 -- 172.21.15.115:0/4138692362 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef000e040 con 0x7f2ef8103680 2023-07-14T18:11:11.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2eef7fe700 1 -- 172.21.15.115:0/4138692362 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2ef0003e10 con 0x7f2ef8103680 2023-07-14T18:11:11.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef8194610 con 0x7f2ef8103680 2023-07-14T18:11:11.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2eef7fe700 1 -- 172.21.15.115:0/4138692362 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef001b940 con 0x7f2ef8103680 2023-07-14T18:11:11.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.605+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef81a0960 con 0x7f2ef8103680 2023-07-14T18:11:11.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.606+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef804e340 con 0x7f2ef8103680 2023-07-14T18:11:11.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.607+0000 7f2eef7fe700 1 -- 172.21.15.115:0/4138692362 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f2ef0005690 con 0x7f2ef8103680 2023-07-14T18:11:11.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.608+0000 7f2eef7fe700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ee40a1260 0x7f2ee40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:11.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.608+0000 7f2efe116700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ee40a1260 0x7f2ee40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:11.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.608+0000 7f2eef7fe700 1 -- 172.21.15.115:0/4138692362 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2ef00ce6d0 con 0x7f2ef8103680 2023-07-14T18:11:11.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.608+0000 7f2efe116700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ee40a1260 0x7f2ee40a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4005950 tx=0x7f2ef400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:11.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.611+0000 7f2eef7fe700 1 -- 172.21.15.115:0/4138692362 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2ef0097b60 con 0x7f2ef8103680 2023-07-14T18:11:11.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.771+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f514c1a2bf0 con 0x7f514c0ff7e0 2023-07-14T18:11:11.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.772+0000 7f513b7fe700 1 -- 172.21.15.115:0/1457889871 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f514009adf0 con 0x7f514c0ff7e0 2023-07-14T18:11:11.812 INFO:teuthology.orchestra.run.smithi115.stdout:249108103202 2023-07-14T18:11:11.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51340a11e0 msgr2=0x7f51340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51340a11e0 0x7f51340a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f513c00ac50 tx=0x7f513c00a380 comp rx=0 tx=0).stop 2023-07-14T18:11:11.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 msgr2=0x7f514c196ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 0x7f514c196ff0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f514000b580 tx=0x7f5140009580 comp rx=0 tx=0).stop 2023-07-14T18:11:11.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 shutdown_connections 2023-07-14T18:11:11.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51340a11e0 0x7f51340a3690 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f514c103ee0 0x7f514c1a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f514c0fee40 0x7f514c196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 --2- 172.21.15.115:0/1457889871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f514c0ff7e0 0x7f514c196ff0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 >> 172.21.15.115:0/1457889871 conn(0x7f514c075ca0 msgr2=0x7f514c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:11.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 shutdown_connections 2023-07-14T18:11:11.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.774+0000 7f5150b1c700 1 -- 172.21.15.115:0/1457889871 wait complete. 2023-07-14T18:11:11.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.816+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2ef8194f00 con 0x7f2ef8103680 2023-07-14T18:11:11.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.816+0000 7f2eef7fe700 1 -- 172.21.15.115:0/4138692362 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2ef009b180 con 0x7f2ef8103680 2023-07-14T18:11:11.856 INFO:teuthology.orchestra.run.smithi115.stdout:137438953523 2023-07-14T18:11:11.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ee40a1260 msgr2=0x7f2ee40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ee40a1260 0x7f2ee40a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4005950 tx=0x7f2ef400b410 comp rx=0 tx=0).stop 2023-07-14T18:11:11.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 msgr2=0x7f2ef81939f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:11.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 0x7f2ef81939f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0009380 tx=0x7f2ef00093b0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 shutdown_connections 2023-07-14T18:11:11.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ee40a1260 0x7f2ee40a3710 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2ef8103680 0x7f2ef81939f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef80fe5e0 0x7f2ef8199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 --2- 172.21.15.115:0/4138692362 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ef80fef80 0x7f2ef8199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:11.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.818+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 >> 172.21.15.115:0/4138692362 conn(0x7f2ef8075280 msgr2=0x7f2ef80fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:11.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.819+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 shutdown_connections 2023-07-14T18:11:11.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:11.819+0000 7f2f0037a700 1 -- 172.21.15.115:0/4138692362 wait complete. 2023-07-14T18:11:11.868 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:12.106 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:11 smithi115 ceph-mon[127451]: pgmap v278: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-07-14T18:11:12.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:11 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1457889871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T18:11:12.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:11 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4138692362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T18:11:12.371 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:11 smithi150 ceph-mon[128853]: pgmap v278: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-07-14T18:11:12.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:11 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1457889871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T18:11:12.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:11 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/4138692362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T18:11:12.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:11 smithi187 ceph-mon[129317]: pgmap v278: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-07-14T18:11:12.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:11 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1457889871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T18:11:12.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:11 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/4138692362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T18:11:12.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.487+0000 7fd2d4877700 1 -- 172.21.15.115:0/2572205497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d00ffdb0 msgr2=0x7fd2d010ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:12.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.487+0000 7fd2d4877700 1 --2- 172.21.15.115:0/2572205497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d00ffdb0 0x7fd2d010ba80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800b0d0 tx=0x7fd2c800b3e0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.487+0000 7fd2d4877700 1 -- 172.21.15.115:0/2572205497 shutdown_connections 2023-07-14T18:11:12.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.487+0000 7fd2d4877700 1 --2- 172.21.15.115:0/2572205497 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d00ffdb0 0x7fd2d010ba80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.487+0000 7fd2d4877700 1 --2- 172.21.15.115:0/2572205497 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d0103be0 0x7fd2d0103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.487+0000 7fd2d4877700 1 --2- 172.21.15.115:0/2572205497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d00ff400 0x7fd2d00ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.487+0000 7fd2d4877700 1 -- 172.21.15.115:0/2572205497 >> 172.21.15.115:0/2572205497 conn(0x7fd2d00751e0 msgr2=0x7fd2d00755e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:12.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.487+0000 7fd2d4877700 1 -- 172.21.15.115:0/2572205497 shutdown_connections 2023-07-14T18:11:12.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.487+0000 7fd2d4877700 1 -- 172.21.15.115:0/2572205497 wait complete. 2023-07-14T18:11:12.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.488+0000 7fd2d4877700 1 Processor -- start 2023-07-14T18:11:12.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.488+0000 7fd2d4877700 1 -- start start 2023-07-14T18:11:12.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.488+0000 7fd2d4877700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d00ff400 0x7fd2d0196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:12.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.488+0000 7fd2d4877700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d00ffdb0 0x7fd2d01971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:12.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.488+0000 7fd2d4877700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d0103be0 0x7fd2d01a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:12.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.488+0000 7fd2d4877700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d0197ab0 con 0x7fd2d0103be0 2023-07-14T18:11:12.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.488+0000 7fd2d4877700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d01a2220 con 0x7fd2d00ff400 2023-07-14T18:11:12.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.488+0000 7fd2d4877700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d01a2390 con 0x7fd2d00ffdb0 2023-07-14T18:11:12.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ced9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d0103be0 0x7fd2d01a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:12.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ce59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d00ff400 0x7fd2d0196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:12.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ce59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d00ff400 0x7fd2d0196c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.115:39048/0 (socket says 172.21.15.115:39048) 2023-07-14T18:11:12.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ce59c700 1 -- 172.21.15.115:0/50000545 learned_addr learned my addr 172.21.15.115:0/50000545 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:12.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ced9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d0103be0 0x7fd2d01a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53398/0 (socket says 172.21.15.115:53398) 2023-07-14T18:11:12.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2cdd9b700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d00ffdb0 0x7fd2d01971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:12.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ced9d700 1 -- 172.21.15.115:0/50000545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d00ffdb0 msgr2=0x7fd2d01971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:12.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ced9d700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d00ffdb0 0x7fd2d01971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ced9d700 1 -- 172.21.15.115:0/50000545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d00ff400 msgr2=0x7fd2d0196c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:12.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ced9d700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d00ff400 0x7fd2d0196c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ced9d700 1 -- 172.21.15.115:0/50000545 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c8009d00 con 0x7fd2d0103be0 2023-07-14T18:11:12.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2ced9d700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d0103be0 0x7fd2d01a1ce0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800b820 tx=0x7fd2c8009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:12.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2bf7fe700 1 -- 172.21.15.115:0/50000545 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c800e040 con 0x7fd2d0103be0 2023-07-14T18:11:12.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d01a2610 con 0x7fd2d0103be0 2023-07-14T18:11:12.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2bf7fe700 1 -- 172.21.15.115:0/50000545 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd2c801dd20 con 0x7fd2d0103be0 2023-07-14T18:11:12.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2bf7fe700 1 -- 172.21.15.115:0/50000545 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c801bc30 con 0x7fd2d0103be0 2023-07-14T18:11:12.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.489+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d01a2ab0 con 0x7fd2d0103be0 2023-07-14T18:11:12.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.490+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d004e340 con 0x7fd2d0103be0 2023-07-14T18:11:12.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.491+0000 7fd2bf7fe700 1 -- 172.21.15.115:0/50000545 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fd2c801d890 con 0x7fd2d0103be0 2023-07-14T18:11:12.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.492+0000 7fd2bf7fe700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2b80a11e0 0x7fd2b80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:12.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.492+0000 7fd2ce59c700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2b80a11e0 0x7fd2b80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:12.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.492+0000 7fd2bf7fe700 1 -- 172.21.15.115:0/50000545 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd2c80ceef0 con 0x7fd2d0103be0 2023-07-14T18:11:12.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.492+0000 7fd2ce59c700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2b80a11e0 0x7fd2b80a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009ad0 tx=0x7fd2c0009290 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:12.539 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.496+0000 7fd2bf7fe700 1 -- 172.21.15.115:0/50000545 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd2c8098380 con 0x7fd2d0103be0 2023-07-14T18:11:12.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.705+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd2d01a2da0 con 0x7fd2d0103be0 2023-07-14T18:11:12.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.705+0000 7fd2bf7fe700 1 -- 172.21.15.115:0/50000545 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd2c8021020 con 0x7fd2d0103be0 2023-07-14T18:11:12.745 INFO:teuthology.orchestra.run.smithi115.stdout:34359738435 2023-07-14T18:11:12.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.707+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2b80a11e0 msgr2=0x7fd2b80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:12.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.707+0000 7fd2d4877700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2b80a11e0 0x7fd2b80a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009ad0 tx=0x7fd2c0009290 comp rx=0 tx=0).stop 2023-07-14T18:11:12.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.707+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d0103be0 msgr2=0x7fd2d01a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:12.747 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.707+0000 7fd2d4877700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d0103be0 0x7fd2d01a1ce0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800b820 tx=0x7fd2c8009380 comp rx=0 tx=0).stop 2023-07-14T18:11:12.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.707+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 shutdown_connections 2023-07-14T18:11:12.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.707+0000 7fd2d4877700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd2b80a11e0 0x7fd2b80a3690 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.707+0000 7fd2d4877700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2d0103be0 0x7fd2d01a1ce0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.707+0000 7fd2d4877700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d00ff400 0x7fd2d0196c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.707+0000 7fd2d4877700 1 --2- 172.21.15.115:0/50000545 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd2d00ffdb0 0x7fd2d01971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:12.748 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.708+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 >> 172.21.15.115:0/50000545 conn(0x7fd2d00751e0 msgr2=0x7fd2d00fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:12.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.708+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 shutdown_connections 2023-07-14T18:11:12.749 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:12.708+0000 7fd2d4877700 1 -- 172.21.15.115:0/50000545 wait complete. 2023-07-14T18:11:13.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 -- 172.21.15.115:0/1228351908 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 msgr2=0x7f0fe410cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 --2- 172.21.15.115:0/1228351908 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 0x7f0fe410cb30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00b0a0 tx=0x7f0fdc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 -- 172.21.15.115:0/1228351908 shutdown_connections 2023-07-14T18:11:13.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 --2- 172.21.15.115:0/1228351908 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 0x7f0fe410cb30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 --2- 172.21.15.115:0/1228351908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe4104c70 0x7f0fe4105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 --2- 172.21.15.115:0/1228351908 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fe40fe210 0x7f0fe40fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 -- 172.21.15.115:0/1228351908 >> 172.21.15.115:0/1228351908 conn(0x7f0fe4075280 msgr2=0x7f0fe4075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:13.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 -- 172.21.15.115:0/1228351908 shutdown_connections 2023-07-14T18:11:13.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 -- 172.21.15.115:0/1228351908 wait complete. 2023-07-14T18:11:13.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 Processor -- start 2023-07-14T18:11:13.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.387+0000 7f0feb18a700 1 -- start start 2023-07-14T18:11:13.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0feb18a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40fe210 0x7f0fe4195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:13.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0feb18a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 0x7f0fe41962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:13.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0feb18a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fe4104c70 0x7f0fe419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:13.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0feb18a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe419ad70 con 0x7f0fe40febc0 2023-07-14T18:11:13.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0feb18a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe419aee0 con 0x7f0fe40fe210 2023-07-14T18:11:13.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0feb18a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe419b0a0 con 0x7f0fe4104c70 2023-07-14T18:11:13.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 0x7f0fe41962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:13.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe8f26700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40fe210 0x7f0fe4195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:13.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 0x7f0fe41962f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53410/0 (socket says 172.21.15.115:53410) 2023-07-14T18:11:13.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe3fff700 1 -- 172.21.15.115:0/886429534 learned_addr learned my addr 172.21.15.115:0/886429534 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:13.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe9727700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fe4104c70 0x7f0fe419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:13.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe3fff700 1 -- 172.21.15.115:0/886429534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fe4104c70 msgr2=0x7f0fe419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe3fff700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fe4104c70 0x7f0fe419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe3fff700 1 -- 172.21.15.115:0/886429534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40fe210 msgr2=0x7f0fe4195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe3fff700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40fe210 0x7f0fe4195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe3fff700 1 -- 172.21.15.115:0/886429534 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc009d00 con 0x7f0fe40febc0 2023-07-14T18:11:13.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe3fff700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 0x7f0fe41962f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000d950 tx=0x7f0fd000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:13.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe1ffb700 1 -- 172.21.15.115:0/886429534 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0009940 con 0x7f0fe40febc0 2023-07-14T18:11:13.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe1ffb700 1 -- 172.21.15.115:0/886429534 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0fd0010460 con 0x7f0fe40febc0 2023-07-14T18:11:13.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe419b380 con 0x7f0fe40febc0 2023-07-14T18:11:13.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.388+0000 7f0fe1ffb700 1 -- 172.21.15.115:0/886429534 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000f620 con 0x7f0fe40febc0 2023-07-14T18:11:13.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.389+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe41a2be0 con 0x7f0fe40febc0 2023-07-14T18:11:13.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.390+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe404e340 con 0x7f0fe40febc0 2023-07-14T18:11:13.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.390+0000 7f0fe1ffb700 1 -- 172.21.15.115:0/886429534 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f0fd00105d0 con 0x7f0fe40febc0 2023-07-14T18:11:13.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.391+0000 7f0fe1ffb700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0fd40a1260 0x7f0fd40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:13.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.391+0000 7f0fe8f26700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0fd40a1260 0x7f0fd40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:13.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.391+0000 7f0fe1ffb700 1 -- 172.21.15.115:0/886429534 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0fd00cca60 con 0x7f0fe40febc0 2023-07-14T18:11:13.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.392+0000 7f0fe8f26700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0fd40a1260 0x7f0fd40a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8005950 tx=0x7f0fd800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:13.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.395+0000 7f0fe1ffb700 1 -- 172.21.15.115:0/886429534 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0fd0095e60 con 0x7f0fe40febc0 2023-07-14T18:11:13.491 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103201 got 249108103202 for osd.8 2023-07-14T18:11:13.492 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:13.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.611+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0fe41a2dd0 con 0x7f0fe40febc0 2023-07-14T18:11:13.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.611+0000 7f0fe1ffb700 1 -- 172.21.15.115:0/886429534 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0fd0016090 con 0x7f0fe40febc0 2023-07-14T18:11:13.651 INFO:teuthology.orchestra.run.smithi115.stdout:55834574910 2023-07-14T18:11:13.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.613+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0fd40a1260 msgr2=0x7f0fd40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.613+0000 7f0feb18a700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0fd40a1260 0x7f0fd40a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8005950 tx=0x7f0fd800b410 comp rx=0 tx=0).stop 2023-07-14T18:11:13.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.613+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 msgr2=0x7f0fe41962f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.613+0000 7f0feb18a700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 0x7f0fe41962f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000d950 tx=0x7f0fd000dc60 comp rx=0 tx=0).stop 2023-07-14T18:11:13.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.614+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 shutdown_connections 2023-07-14T18:11:13.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.614+0000 7f0feb18a700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0fd40a1260 0x7f0fd40a3710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.614+0000 7f0feb18a700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0fe4104c70 0x7f0fe419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.614+0000 7f0feb18a700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40fe210 0x7f0fe4195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.614+0000 7f0feb18a700 1 --2- 172.21.15.115:0/886429534 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0fe40febc0 0x7f0fe41962f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.614+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 >> 172.21.15.115:0/886429534 conn(0x7f0fe4075280 msgr2=0x7f0fe40fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:13.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.614+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 shutdown_connections 2023-07-14T18:11:13.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.614+0000 7f0feb18a700 1 -- 172.21.15.115:0/886429534 wait complete. 2023-07-14T18:11:13.723 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:13 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/50000545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T18:11:13.723 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:13 smithi115 ceph-mon[127451]: pgmap v279: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2023-07-14T18:11:13.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.585+0000 7fd40f182700 1 -- 172.21.15.115:0/3416062777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd40810a270 msgr2=0x7fd40810c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.585+0000 7fd40f182700 1 --2- 172.21.15.115:0/3416062777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd40810a270 0x7fd40810c6b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd40000b0a0 tx=0x7fd40000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.754 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.585+0000 7fd40f182700 1 -- 172.21.15.115:0/3416062777 shutdown_connections 2023-07-14T18:11:13.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.585+0000 7fd40f182700 1 --2- 172.21.15.115:0/3416062777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd40810a270 0x7fd40810c6b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.585+0000 7fd40f182700 1 --2- 172.21.15.115:0/3416062777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd408100f20 0x7fd4081012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.585+0000 7fd40f182700 1 --2- 172.21.15.115:0/3416062777 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd408101830 0x7fd408109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.585+0000 7fd40f182700 1 -- 172.21.15.115:0/3416062777 >> 172.21.15.115:0/3416062777 conn(0x7fd408075ca0 msgr2=0x7fd4080760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:13.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.585+0000 7fd40f182700 1 -- 172.21.15.115:0/3416062777 shutdown_connections 2023-07-14T18:11:13.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.585+0000 7fd40f182700 1 -- 172.21.15.115:0/3416062777 wait complete. 2023-07-14T18:11:13.755 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40f182700 1 Processor -- start 2023-07-14T18:11:13.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40f182700 1 -- start start 2023-07-14T18:11:13.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40f182700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd408100f20 0x7fd408195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:13.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40f182700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd408101830 0x7fd408196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:13.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40f182700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd40810a270 0x7fd40819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:13.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40f182700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40819adc0 con 0x7fd408100f20 2023-07-14T18:11:13.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40f182700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40819af30 con 0x7fd408101830 2023-07-14T18:11:13.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40f182700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40819b0f0 con 0x7fd40810a270 2023-07-14T18:11:13.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40cf1e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd408100f20 0x7fd408195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:13.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40cf1e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd408100f20 0x7fd408195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53422/0 (socket says 172.21.15.115:53422) 2023-07-14T18:11:13.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd40cf1e700 1 -- 172.21.15.115:0/291255122 learned_addr learned my addr 172.21.15.115:0/291255122 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:13.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.586+0000 7fd407fff700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd408101830 0x7fd408196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:13.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd40cf1e700 1 -- 172.21.15.115:0/291255122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd40810a270 msgr2=0x7fd40819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd40cf1e700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd40810a270 0x7fd40819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd40cf1e700 1 -- 172.21.15.115:0/291255122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd408101830 msgr2=0x7fd408196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd40cf1e700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd408101830 0x7fd408196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd40cf1e700 1 -- 172.21.15.115:0/291255122 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd400009d00 con 0x7fd408100f20 2023-07-14T18:11:13.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd40cf1e700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd408100f20 0x7fd408195e00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c920 tx=0x7fd3fc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:13.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd405ffb700 1 -- 172.21.15.115:0/291255122 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc007b00 con 0x7fd408100f20 2023-07-14T18:11:13.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd405ffb700 1 -- 172.21.15.115:0/291255122 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd3fc004cc0 con 0x7fd408100f20 2023-07-14T18:11:13.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40819b3d0 con 0x7fd408100f20 2023-07-14T18:11:13.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd405ffb700 1 -- 172.21.15.115:0/291255122 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc005590 con 0x7fd408100f20 2023-07-14T18:11:13.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.587+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4081a2c30 con 0x7fd408100f20 2023-07-14T18:11:13.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.588+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40804e340 con 0x7fd408100f20 2023-07-14T18:11:13.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.589+0000 7fd405ffb700 1 -- 172.21.15.115:0/291255122 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fd3fc004750 con 0x7fd408100f20 2023-07-14T18:11:13.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.590+0000 7fd405ffb700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3f80a11e0 0x7fd3f80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:13.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.590+0000 7fd407fff700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3f80a11e0 0x7fd3f80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:13.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.590+0000 7fd405ffb700 1 -- 172.21.15.115:0/291255122 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd3fc0cc8d0 con 0x7fd408100f20 2023-07-14T18:11:13.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.590+0000 7fd407fff700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3f80a11e0 0x7fd3f80a3690 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4005950 tx=0x7fd3f40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:13.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.594+0000 7fd405ffb700 1 -- 172.21.15.115:0/291255122 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd3fc095cd0 con 0x7fd408100f20 2023-07-14T18:11:13.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.810+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd4081a2e20 con 0x7fd408100f20 2023-07-14T18:11:13.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.810+0000 7fd405ffb700 1 -- 172.21.15.115:0/291255122 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd3fc01d090 con 0x7fd408100f20 2023-07-14T18:11:13.850 INFO:teuthology.orchestra.run.smithi115.stdout:193273528362 2023-07-14T18:11:13.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3f80a11e0 msgr2=0x7fd3f80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3f80a11e0 0x7fd3f80a3690 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4005950 tx=0x7fd3f40058e0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd408100f20 msgr2=0x7fd408195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:13.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd408100f20 0x7fd408195e00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c920 tx=0x7fd3fc00cce0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 shutdown_connections 2023-07-14T18:11:13.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3f80a11e0 0x7fd3f80a3690 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd40810a270 0x7fd40819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd408100f20 0x7fd408195e00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 --2- 172.21.15.115:0/291255122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd408101830 0x7fd408196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:13.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.812+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 >> 172.21.15.115:0/291255122 conn(0x7fd408075ca0 msgr2=0x7fd40810ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:13.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.813+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 shutdown_connections 2023-07-14T18:11:13.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.813+0000 7fd40f182700 1 -- 172.21.15.115:0/291255122 wait complete. 2023-07-14T18:11:13.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:13 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/50000545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T18:11:13.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:13 smithi150 ceph-mon[128853]: pgmap v279: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2023-07-14T18:11:13.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:13 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/50000545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T18:11:13.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:13 smithi187 ceph-mon[129317]: pgmap v279: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2023-07-14T18:11:14.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.709+0000 7f06ed2dc700 1 -- 172.21.15.115:0/4157948760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e810a300 msgr2=0x7f06e810c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.709+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/4157948760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e810a300 0x7f06e810c760 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f06e000b0a0 tx=0x7f06e000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.709+0000 7f06ed2dc700 1 -- 172.21.15.115:0/4157948760 shutdown_connections 2023-07-14T18:11:14.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.709+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/4157948760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e810a300 0x7f06e810c760 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.709+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/4157948760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e8100f20 0x7f06e81012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.709+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/4157948760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e8101830 0x7f06e8109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.709+0000 7f06ed2dc700 1 -- 172.21.15.115:0/4157948760 >> 172.21.15.115:0/4157948760 conn(0x7f06e80fa720 msgr2=0x7f06e80fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:14.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.709+0000 7f06ed2dc700 1 -- 172.21.15.115:0/4157948760 shutdown_connections 2023-07-14T18:11:14.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.709+0000 7f06ed2dc700 1 -- 172.21.15.115:0/4157948760 wait complete. 2023-07-14T18:11:14.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06ed2dc700 1 Processor -- start 2023-07-14T18:11:14.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06ed2dc700 1 -- start start 2023-07-14T18:11:14.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06ed2dc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e8100f20 0x7f06e8195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:14.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06ed2dc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e8101830 0x7f06e8196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:14.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06ed2dc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e810a300 0x7f06e819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:14.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06ed2dc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e819adc0 con 0x7f06e8100f20 2023-07-14T18:11:14.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06ed2dc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e819af30 con 0x7f06e810a300 2023-07-14T18:11:14.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06ed2dc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e819b0f0 con 0x7f06e8101830 2023-07-14T18:11:14.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06e6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e8100f20 0x7f06e8195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:14.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06e6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e8100f20 0x7f06e8195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53436/0 (socket says 172.21.15.115:53436) 2023-07-14T18:11:14.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06e6ffd700 1 -- 172.21.15.115:0/2439987473 learned_addr learned my addr 172.21.15.115:0/2439987473 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:14.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06e67fc700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e8101830 0x7f06e8196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:14.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.710+0000 7f06e77fe700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e810a300 0x7f06e819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:14.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06e6ffd700 1 -- 172.21.15.115:0/2439987473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e8101830 msgr2=0x7f06e8196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06e6ffd700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e8101830 0x7f06e8196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06e6ffd700 1 -- 172.21.15.115:0/2439987473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e810a300 msgr2=0x7f06e819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06e6ffd700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e810a300 0x7f06e819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06e6ffd700 1 -- 172.21.15.115:0/2439987473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e0009d00 con 0x7f06e8100f20 2023-07-14T18:11:14.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06e77fe700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e810a300 0x7f06e819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:14.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06e6ffd700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e8100f20 0x7f06e8195e00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06d800e9a0 tx=0x7f06d800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:14.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06cffff700 1 -- 172.21.15.115:0/2439987473 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8009790 con 0x7f06e8100f20 2023-07-14T18:11:14.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e819b3d0 con 0x7f06e8100f20 2023-07-14T18:11:14.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06cffff700 1 -- 172.21.15.115:0/2439987473 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f06d800cbd0 con 0x7f06e8100f20 2023-07-14T18:11:14.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06cffff700 1 -- 172.21.15.115:0/2439987473 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d8018680 con 0x7f06e8100f20 2023-07-14T18:11:14.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.711+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e80fec80 con 0x7f06e8100f20 2023-07-14T18:11:14.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.713+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e80682d0 con 0x7f06e8100f20 2023-07-14T18:11:14.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.713+0000 7f06cffff700 1 -- 172.21.15.115:0/2439987473 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f06d800cd40 con 0x7f06e8100f20 2023-07-14T18:11:14.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.714+0000 7f06cffff700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f06d00a11e0 0x7f06d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:14.185 INFO:teuthology.orchestra.run.smithi115.stdout:283467841567 2023-07-14T18:11:14.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.714+0000 7f06e67fc700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f06d00a11e0 0x7f06d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:14.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.714+0000 7f06cffff700 1 -- 172.21.15.115:0/2439987473 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f06d8014070 con 0x7f06e8100f20 2023-07-14T18:11:14.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.714+0000 7f06e67fc700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f06d00a11e0 0x7f06d00a3690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f06dc005950 tx=0x7f06dc0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:14.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.718+0000 7f06cffff700 1 -- 172.21.15.115:0/2439987473 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f06d8051780 con 0x7f06e8100f20 2023-07-14T18:11:14.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.923+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f06e8197160 con 0x7f06e8100f20 2023-07-14T18:11:14.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.923+0000 7f06cffff700 1 -- 172.21.15.115:0/2439987473 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f06d8099e60 con 0x7f06e8100f20 2023-07-14T18:11:14.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f06d00a11e0 msgr2=0x7f06d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f06d00a11e0 0x7f06d00a3690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f06dc005950 tx=0x7f06dc0058e0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e8100f20 msgr2=0x7f06e8195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e8100f20 0x7f06e8195e00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06d800e9a0 tx=0x7f06d800ed60 comp rx=0 tx=0).stop 2023-07-14T18:11:14.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 shutdown_connections 2023-07-14T18:11:14.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f06d00a11e0 0x7f06d00a3690 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e810a300 0x7f06e819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06e8100f20 0x7f06e8195e00 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 --2- 172.21.15.115:0/2439987473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e8101830 0x7f06e8196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 >> 172.21.15.115:0/2439987473 conn(0x7f06e80fa720 msgr2=0x7f06e810c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:14.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 shutdown_connections 2023-07-14T18:11:14.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:13.925+0000 7f06ed2dc700 1 -- 172.21.15.115:0/2439987473 wait complete. 2023-07-14T18:11:14.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.384+0000 7fa25fe0a700 1 -- 172.21.15.115:0/595822561 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2581007a0 msgr2=0x7fa258100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.384+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/595822561 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2581007a0 0x7fa258100c10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0099c0 tx=0x7fa24c009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.384+0000 7fa25fe0a700 1 -- 172.21.15.115:0/595822561 shutdown_connections 2023-07-14T18:11:14.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.384+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/595822561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa258101150 0x7fa25810bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.384+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/595822561 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2580fef40 0x7fa2580ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.384+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/595822561 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2581007a0 0x7fa258100c10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.384+0000 7fa25fe0a700 1 -- 172.21.15.115:0/595822561 >> 172.21.15.115:0/595822561 conn(0x7fa258075c90 msgr2=0x7fa258076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:14.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.384+0000 7fa25fe0a700 1 -- 172.21.15.115:0/595822561 shutdown_connections 2023-07-14T18:11:14.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.384+0000 7fa25fe0a700 1 -- 172.21.15.115:0/595822561 wait complete. 2023-07-14T18:11:14.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25fe0a700 1 Processor -- start 2023-07-14T18:11:14.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25fe0a700 1 -- start start 2023-07-14T18:11:14.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25fe0a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2580fef40 0x7fa258194cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:14.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25fe0a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2581007a0 0x7fa2581951f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:14.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25fe0a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa258101150 0x7fa2581985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:14.447 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25fe0a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa258198bb0 con 0x7fa258101150 2023-07-14T18:11:14.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25fe0a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa258198d20 con 0x7fa2581007a0 2023-07-14T18:11:14.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25fe0a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa258198ee0 con 0x7fa2580fef40 2023-07-14T18:11:14.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25e3a7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa258101150 0x7fa2581985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:14.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25e3a7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa258101150 0x7fa2581985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53454/0 (socket says 172.21.15.115:53454) 2023-07-14T18:11:14.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25e3a7700 1 -- 172.21.15.115:0/2670892180 learned_addr learned my addr 172.21.15.115:0/2670892180 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:14.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25dba6700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2580fef40 0x7fa258194cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:14.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.385+0000 7fa25d3a5700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2581007a0 0x7fa2581951f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:14.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa25e3a7700 1 -- 172.21.15.115:0/2670892180 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2580fef40 msgr2=0x7fa258194cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa25e3a7700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2580fef40 0x7fa258194cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa25e3a7700 1 -- 172.21.15.115:0/2670892180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2581007a0 msgr2=0x7fa2581951f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa25e3a7700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2581007a0 0x7fa2581951f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa25e3a7700 1 -- 172.21.15.115:0/2670892180 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c0095f0 con 0x7fa258101150 2023-07-14T18:11:14.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa25e3a7700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa258101150 0x7fa2581985b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa25000b970 tx=0x7fa25000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:14.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa24affd700 1 -- 172.21.15.115:0/2670892180 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000c930 con 0x7fa258101150 2023-07-14T18:11:14.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa24affd700 1 -- 172.21.15.115:0/2670892180 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa250014440 con 0x7fa258101150 2023-07-14T18:11:14.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa24affd700 1 -- 172.21.15.115:0/2670892180 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2500075d0 con 0x7fa258101150 2023-07-14T18:11:14.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2581991c0 con 0x7fa258101150 2023-07-14T18:11:14.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.386+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2581a1b10 con 0x7fa258101150 2023-07-14T18:11:14.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.387+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25804e340 con 0x7fa258101150 2023-07-14T18:11:14.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.388+0000 7fa24affd700 1 -- 172.21.15.115:0/2670892180 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fa2500145b0 con 0x7fa258101150 2023-07-14T18:11:14.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.389+0000 7fa24affd700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa2440a11e0 0x7fa2440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:14.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.389+0000 7fa25dba6700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa2440a11e0 0x7fa2440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:14.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.389+0000 7fa24affd700 1 -- 172.21.15.115:0/2670892180 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa250006e40 con 0x7fa258101150 2023-07-14T18:11:14.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.389+0000 7fa25dba6700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa2440a11e0 0x7fa2440a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa254005950 tx=0x7fa25400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:14.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.393+0000 7fa24affd700 1 -- 172.21.15.115:0/2670892180 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa250095960 con 0x7fa258101150 2023-07-14T18:11:14.526 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:14 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/886429534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T18:11:14.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:14 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/291255122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T18:11:14.527 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:14 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2439987473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T18:11:14.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.598+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fa2581a1ca0 con 0x7fa258101150 2023-07-14T18:11:14.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.598+0000 7fa24affd700 1 -- 172.21.15.115:0/2670892180 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fa250098f80 con 0x7fa258101150 2023-07-14T18:11:14.638 INFO:teuthology.orchestra.run.smithi115.stdout:304942678042 2023-07-14T18:11:14.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa2440a11e0 msgr2=0x7fa2440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa2440a11e0 0x7fa2440a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa254005950 tx=0x7fa25400a300 comp rx=0 tx=0).stop 2023-07-14T18:11:14.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa258101150 msgr2=0x7fa2581985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:14.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa258101150 0x7fa2581985b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa25000b970 tx=0x7fa25000bc80 comp rx=0 tx=0).stop 2023-07-14T18:11:14.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 shutdown_connections 2023-07-14T18:11:14.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa2440a11e0 0x7fa2440a3690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa258101150 0x7fa2581985b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2580fef40 0x7fa258194cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 --2- 172.21.15.115:0/2670892180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2581007a0 0x7fa2581951f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:14.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 >> 172.21.15.115:0/2670892180 conn(0x7fa258075c90 msgr2=0x7fa2580fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:14.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 shutdown_connections 2023-07-14T18:11:14.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:14.600+0000 7fa25fe0a700 1 -- 172.21.15.115:0/2670892180 wait complete. 2023-07-14T18:11:14.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:14 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/886429534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T18:11:14.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:14 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/291255122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T18:11:14.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:14 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2439987473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T18:11:14.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:14 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/886429534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T18:11:14.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:14 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/291255122' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T18:11:14.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:14 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2439987473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T18:11:15.261 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841565 got 283467841567 for osd.9 2023-07-14T18:11:15.261 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:15.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:15 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2670892180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T18:11:15.856 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:15 smithi115 ceph-mon[127451]: pgmap v280: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T18:11:15.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:15 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2670892180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T18:11:15.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:15 smithi150 ceph-mon[128853]: pgmap v280: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T18:11:15.876 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574910 for osd.1 2023-07-14T18:11:15.877 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:15.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:15 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2670892180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T18:11:15.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:15 smithi187 ceph-mon[129317]: pgmap v280: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T18:11:16.036 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953522 got 137438953523 for osd.4 2023-07-14T18:11:16.037 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:16.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.128+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/510055984 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a48101150 msgr2=0x7f8a4810bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.128+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/510055984 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a48101150 0x7f8a4810bb40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000b0d0 tx=0x7f8a4000b3e0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.128+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/510055984 shutdown_connections 2023-07-14T18:11:16.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.128+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/510055984 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a48101150 0x7f8a4810bb40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.128+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/510055984 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a480fef40 0x7f8a480ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.128+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/510055984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a481007a0 0x7f8a48100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.128+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/510055984 >> 172.21.15.115:0/510055984 conn(0x7f8a48075c90 msgr2=0x7f8a48076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:16.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.128+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/510055984 shutdown_connections 2023-07-14T18:11:16.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.128+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/510055984 wait complete. 2023-07-14T18:11:16.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a4c9a6700 1 Processor -- start 2023-07-14T18:11:16.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a4c9a6700 1 -- start start 2023-07-14T18:11:16.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a4c9a6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a480fef40 0x7f8a48195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:16.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a4c9a6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a481007a0 0x7f8a481963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:16.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a4c9a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a48101150 0x7f8a4819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:16.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a4c9a6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4819ae40 con 0x7f8a481007a0 2023-07-14T18:11:16.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a4c9a6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4819afb0 con 0x7f8a480fef40 2023-07-14T18:11:16.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a4c9a6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4819b170 con 0x7f8a48101150 2023-07-14T18:11:16.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a45d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a481007a0 0x7f8a481963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:16.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a45d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a481007a0 0x7f8a481963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53466/0 (socket says 172.21.15.115:53466) 2023-07-14T18:11:16.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a45d9b700 1 -- 172.21.15.115:0/1704554156 learned_addr learned my addr 172.21.15.115:0/1704554156 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:16.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.129+0000 7f8a4659c700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a480fef40 0x7f8a48195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:16.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a45d9b700 1 -- 172.21.15.115:0/1704554156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a48101150 msgr2=0x7f8a4819a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a45d9b700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a48101150 0x7f8a4819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a45d9b700 1 -- 172.21.15.115:0/1704554156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a480fef40 msgr2=0x7f8a48195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a45d9b700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a480fef40 0x7f8a48195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a45d9b700 1 -- 172.21.15.115:0/1704554156 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a40009d00 con 0x7f8a481007a0 2023-07-14T18:11:16.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a45d9b700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a481007a0 0x7f8a481963c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800d950 tx=0x7f8a3800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:16.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a377fe700 1 -- 172.21.15.115:0/1704554156 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a38009940 con 0x7f8a481007a0 2023-07-14T18:11:16.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4819b450 con 0x7f8a481007a0 2023-07-14T18:11:16.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a377fe700 1 -- 172.21.15.115:0/1704554156 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8a38010460 con 0x7f8a481007a0 2023-07-14T18:11:16.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a377fe700 1 -- 172.21.15.115:0/1704554156 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3800f620 con 0x7f8a481007a0 2023-07-14T18:11:16.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.130+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a481a2d30 con 0x7f8a481007a0 2023-07-14T18:11:16.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.131+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4804e340 con 0x7f8a481007a0 2023-07-14T18:11:16.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.132+0000 7f8a377fe700 1 -- 172.21.15.115:0/1704554156 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f8a380105d0 con 0x7f8a481007a0 2023-07-14T18:11:16.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.132+0000 7f8a377fe700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a300a11e0 0x7f8a300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:16.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.133+0000 7f8a4659c700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a300a11e0 0x7f8a300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:16.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.133+0000 7f8a377fe700 1 -- 172.21.15.115:0/1704554156 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8a380ccae0 con 0x7f8a481007a0 2023-07-14T18:11:16.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.133+0000 7f8a4659c700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a300a11e0 0x7f8a300a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009a10 tx=0x7f8a3c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:16.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.137+0000 7f8a377fe700 1 -- 172.21.15.115:0/1704554156 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a38095ee0 con 0x7f8a481007a0 2023-07-14T18:11:16.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.332+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f8a481a2f20 con 0x7f8a481007a0 2023-07-14T18:11:16.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.333+0000 7f8a377fe700 1 -- 172.21.15.115:0/1704554156 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f8a38016090 con 0x7f8a481007a0 2023-07-14T18:11:16.373 INFO:teuthology.orchestra.run.smithi115.stdout:335007449110 2023-07-14T18:11:16.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a300a11e0 msgr2=0x7f8a300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a300a11e0 0x7f8a300a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009a10 tx=0x7f8a3c009190 comp rx=0 tx=0).stop 2023-07-14T18:11:16.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a481007a0 msgr2=0x7f8a481963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a481007a0 0x7f8a481963c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800d950 tx=0x7f8a3800dc60 comp rx=0 tx=0).stop 2023-07-14T18:11:16.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 shutdown_connections 2023-07-14T18:11:16.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8a300a11e0 0x7f8a300a3690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8a48101150 0x7f8a4819a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a480fef40 0x7f8a48195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 --2- 172.21.15.115:0/1704554156 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a481007a0 0x7f8a481963c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 >> 172.21.15.115:0/1704554156 conn(0x7f8a48075c90 msgr2=0x7f8a480689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:16.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 shutdown_connections 2023-07-14T18:11:16.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.335+0000 7f8a4c9a6700 1 -- 172.21.15.115:0/1704554156 wait complete. 2023-07-14T18:11:16.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.412+0000 7fad432c9700 1 -- 172.21.15.115:0/3238766568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c104d10 msgr2=0x7fad3c1050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.412+0000 7fad432c9700 1 --2- 172.21.15.115:0/3238766568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c104d10 0x7fad3c1050e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fad380099c0 tx=0x7fad38009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.412+0000 7fad432c9700 1 -- 172.21.15.115:0/3238766568 shutdown_connections 2023-07-14T18:11:16.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.412+0000 7fad432c9700 1 --2- 172.21.15.115:0/3238766568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad3c0ffbe0 0x7fad3c10ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.412+0000 7fad432c9700 1 --2- 172.21.15.115:0/3238766568 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c104d10 0x7fad3c1050e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.412+0000 7fad432c9700 1 --2- 172.21.15.115:0/3238766568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad3c105620 0x7fad3c0ff4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.412+0000 7fad432c9700 1 -- 172.21.15.115:0/3238766568 >> 172.21.15.115:0/3238766568 conn(0x7fad3c076370 msgr2=0x7fad3c076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:16.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.412+0000 7fad432c9700 1 -- 172.21.15.115:0/3238766568 shutdown_connections 2023-07-14T18:11:16.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad432c9700 1 -- 172.21.15.115:0/3238766568 wait complete. 2023-07-14T18:11:16.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad432c9700 1 Processor -- start 2023-07-14T18:11:16.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad432c9700 1 -- start start 2023-07-14T18:11:16.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad432c9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c0ffbe0 0x7fad3c196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:16.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad432c9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad3c104d10 0x7fad3c197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:16.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad432c9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad3c197730 0x7fad3c1a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:16.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad432c9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c1a23c0 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad432c9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c1a2530 con 0x7fad3c104d10 2023-07-14T18:11:16.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad432c9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c1a26f0 con 0x7fad3c197730 2023-07-14T18:11:16.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.413+0000 7fad41065700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c0ffbe0 0x7fad3c196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:16.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad41065700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c0ffbe0 0x7fad3c196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53478/0 (socket says 172.21.15.115:53478) 2023-07-14T18:11:16.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad41065700 1 -- 172.21.15.115:0/227033494 learned_addr learned my addr 172.21.15.115:0/227033494 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:16.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad40864700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad3c104d10 0x7fad3c197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:16.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad41866700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad3c197730 0x7fad3c1a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:16.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad41065700 1 -- 172.21.15.115:0/227033494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad3c197730 msgr2=0x7fad3c1a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad41065700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad3c197730 0x7fad3c1a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad41065700 1 -- 172.21.15.115:0/227033494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad3c104d10 msgr2=0x7fad3c197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad41065700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad3c104d10 0x7fad3c197090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad41065700 1 -- 172.21.15.115:0/227033494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad380095f0 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad40864700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad3c104d10 0x7fad3c197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:16.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad41065700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c0ffbe0 0x7fad3c196b50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fad38009f50 tx=0x7fad380047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:16.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad327fc700 1 -- 172.21.15.115:0/227033494 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3801d070 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad327fc700 1 -- 172.21.15.115:0/227033494 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fad38004040 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad327fc700 1 -- 172.21.15.115:0/227033494 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38022da0 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c1a2970 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.414+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c1a2e10 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.415+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c04e340 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.417+0000 7fad327fc700 1 -- 172.21.15.115:0/227033494 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fad38022570 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.418+0000 7fad327fc700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad280a1260 0x7fad280a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:16.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.418+0000 7fad327fc700 1 -- 172.21.15.115:0/227033494 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fad380ce020 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.418+0000 7fad40864700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad280a1260 0x7fad280a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:16.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.418+0000 7fad40864700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad280a1260 0x7fad280a3710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fad2c005e10 tx=0x7fad2c005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:16.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.423+0000 7fad327fc700 1 -- 172.21.15.115:0/227033494 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fad380974b0 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.619+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fad3c0fe9b0 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.620+0000 7fad327fc700 1 -- 172.21.15.115:0/227033494 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fad3809aad0 con 0x7fad3c0ffbe0 2023-07-14T18:11:16.659 INFO:teuthology.orchestra.run.smithi115.stdout:158913790000 2023-07-14T18:11:16.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad280a1260 msgr2=0x7fad280a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad280a1260 0x7fad280a3710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fad2c005e10 tx=0x7fad2c005d60 comp rx=0 tx=0).stop 2023-07-14T18:11:16.661 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c0ffbe0 msgr2=0x7fad3c196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:16.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c0ffbe0 0x7fad3c196b50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fad38009f50 tx=0x7fad380047c0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 shutdown_connections 2023-07-14T18:11:16.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fad280a1260 0x7fad280a3710 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad3c197730 0x7fad3c1a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fad3c0ffbe0 0x7fad3c196b50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 --2- 172.21.15.115:0/227033494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad3c104d10 0x7fad3c197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:16.662 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 >> 172.21.15.115:0/227033494 conn(0x7fad3c076370 msgr2=0x7fad3c1029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:16.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 shutdown_connections 2023-07-14T18:11:16.663 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:16.622+0000 7fad432c9700 1 -- 172.21.15.115:0/227033494 wait complete. 2023-07-14T18:11:16.689 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:16 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1704554156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T18:11:16.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:16 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1704554156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T18:11:16.908 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:16 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1704554156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T18:11:17.386 INFO:tasks.cephadm.ceph_manager.ceph:need seq 335007449107 got 335007449110 for osd.11 2023-07-14T18:11:17.386 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:17.492 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678040 got 304942678042 for osd.10 2023-07-14T18:11:17.493 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:17.518 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738433 got 34359738435 for osd.0 2023-07-14T18:11:17.518 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:17.616 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789997 got 158913790000 for osd.5 2023-07-14T18:11:17.617 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:17.642 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:17 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/227033494' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T18:11:17.642 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:17 smithi115 ceph-mon[127451]: pgmap v281: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T18:11:17.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528360 got 193273528362 for osd.6 2023-07-14T18:11:17.758 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:17.770 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:17.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:17 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/227033494' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T18:11:17.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:17 smithi150 ceph-mon[128853]: pgmap v281: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T18:11:17.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:17 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/227033494' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T18:11:17.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:17 smithi187 ceph-mon[129317]: pgmap v281: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T18:11:18.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.112+0000 7f425f67e700 1 -- 172.21.15.115:0/1218054827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580febc0 msgr2=0x7f425810cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:18.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.112+0000 7f425f67e700 1 --2- 172.21.15.115:0/1218054827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580febc0 0x7f425810cb30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f425000b0a0 tx=0x7f425000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.152 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.112+0000 7f425f67e700 1 -- 172.21.15.115:0/1218054827 shutdown_connections 2023-07-14T18:11:18.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.112+0000 7f425f67e700 1 --2- 172.21.15.115:0/1218054827 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580febc0 0x7f425810cb30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.112+0000 7f425f67e700 1 --2- 172.21.15.115:0/1218054827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4258104c70 0x7f4258105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.112+0000 7f425f67e700 1 --2- 172.21.15.115:0/1218054827 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f42580fe210 0x7f42580fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.112+0000 7f425f67e700 1 -- 172.21.15.115:0/1218054827 >> 172.21.15.115:0/1218054827 conn(0x7f4258075280 msgr2=0x7f4258075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:18.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.112+0000 7f425f67e700 1 -- 172.21.15.115:0/1218054827 shutdown_connections 2023-07-14T18:11:18.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.112+0000 7f425f67e700 1 -- 172.21.15.115:0/1218054827 wait complete. 2023-07-14T18:11:18.153 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425f67e700 1 Processor -- start 2023-07-14T18:11:18.154 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425f67e700 1 -- start start 2023-07-14T18:11:18.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425f67e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580fe210 0x7f42580ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:18.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425f67e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42580febc0 0x7f42580ff930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:18.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425f67e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4258104c70 0x7f42580ffeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:18.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425f67e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42581a4240 con 0x7f42580fe210 2023-07-14T18:11:18.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425f67e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42581a43b0 con 0x7f42580febc0 2023-07-14T18:11:18.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425f67e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42581a4570 con 0x7f4258104c70 2023-07-14T18:11:18.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425d41a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580fe210 0x7f42580ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:18.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425d41a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580fe210 0x7f42580ff3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53492/0 (socket says 172.21.15.115:53492) 2023-07-14T18:11:18.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425d41a700 1 -- 172.21.15.115:0/919286110 learned_addr learned my addr 172.21.15.115:0/919286110 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:18.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425cc19700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42580febc0 0x7f42580ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:18.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425dc1b700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4258104c70 0x7f42580ffeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:18.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425d41a700 1 -- 172.21.15.115:0/919286110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4258104c70 msgr2=0x7f42580ffeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:18.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425d41a700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4258104c70 0x7f42580ffeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425d41a700 1 -- 172.21.15.115:0/919286110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42580febc0 msgr2=0x7f42580ff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:18.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425d41a700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42580febc0 0x7f42580ff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.157 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425d41a700 1 -- 172.21.15.115:0/919286110 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250009d00 con 0x7f42580fe210 2023-07-14T18:11:18.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.113+0000 7f425d41a700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580fe210 0x7f42580ff3f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f425400d950 tx=0x7f425400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:18.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.114+0000 7f424e7fc700 1 -- 172.21.15.115:0/919286110 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425400f890 con 0x7f42580fe210 2023-07-14T18:11:18.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.114+0000 7f424e7fc700 1 -- 172.21.15.115:0/919286110 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f425400fe80 con 0x7f42580fe210 2023-07-14T18:11:18.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.114+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42581a4850 con 0x7f42580fe210 2023-07-14T18:11:18.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.114+0000 7f424e7fc700 1 -- 172.21.15.115:0/919286110 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425400e3b0 con 0x7f42580fe210 2023-07-14T18:11:18.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.114+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42581a4da0 con 0x7f42580fe210 2023-07-14T18:11:18.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.115+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425804e340 con 0x7f42580fe210 2023-07-14T18:11:18.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.116+0000 7f424e7fc700 1 -- 172.21.15.115:0/919286110 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f4254010460 con 0x7f42580fe210 2023-07-14T18:11:18.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.117+0000 7f424e7fc700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42440a1260 0x7f42440a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:18.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.117+0000 7f425cc19700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42440a1260 0x7f42440a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:18.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.117+0000 7f424e7fc700 1 -- 172.21.15.115:0/919286110 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f42540cc980 con 0x7f42580fe210 2023-07-14T18:11:18.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.117+0000 7f425cc19700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42440a1260 0x7f42440a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4258100be0 tx=0x7f424800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:18.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.120+0000 7f424e7fc700 1 -- 172.21.15.115:0/919286110 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4254095d80 con 0x7f42580fe210 2023-07-14T18:11:18.364 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.325+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f42581a4fc0 con 0x7f42580fe210 2023-07-14T18:11:18.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.325+0000 7f424e7fc700 1 -- 172.21.15.115:0/919286110 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4254020090 con 0x7f42580fe210 2023-07-14T18:11:18.365 INFO:teuthology.orchestra.run.smithi115.stdout:214748364839 2023-07-14T18:11:18.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.327+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42440a1260 msgr2=0x7f42440a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:18.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.327+0000 7f425f67e700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42440a1260 0x7f42440a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4258100be0 tx=0x7f424800a300 comp rx=0 tx=0).stop 2023-07-14T18:11:18.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.327+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580fe210 msgr2=0x7f42580ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:18.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.327+0000 7f425f67e700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580fe210 0x7f42580ff3f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f425400d950 tx=0x7f425400dc60 comp rx=0 tx=0).stop 2023-07-14T18:11:18.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.328+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 shutdown_connections 2023-07-14T18:11:18.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.328+0000 7f425f67e700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f42440a1260 0x7f42440a3710 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.328+0000 7f425f67e700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4258104c70 0x7f42580ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.328+0000 7f425f67e700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f42580fe210 0x7f42580ff3f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.328+0000 7f425f67e700 1 --2- 172.21.15.115:0/919286110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42580febc0 0x7f42580ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:18.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.328+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 >> 172.21.15.115:0/919286110 conn(0x7f4258075280 msgr2=0x7f42580fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:18.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.328+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 shutdown_connections 2023-07-14T18:11:18.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:18.328+0000 7f425f67e700 1 -- 172.21.15.115:0/919286110 wait complete. 2023-07-14T18:11:18.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:18 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:11:18.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:18 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/919286110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T18:11:18.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:18 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:11:18.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:18 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/919286110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T18:11:18.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:18 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:11:18.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:18 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/919286110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T18:11:18.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364837 got 214748364839 for osd.7 2023-07-14T18:11:18.892 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:19.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:19 smithi115 ceph-mon[127451]: pgmap v282: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T18:11:19.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.592+0000 7faad6e39700 1 -- 172.21.15.115:0/3890467221 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad0100950 msgr2=0x7faad01048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:19.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.592+0000 7faad6e39700 1 --2- 172.21.15.115:0/3890467221 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad0100950 0x7faad01048d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faabc0099c0 tx=0x7faabc009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.592+0000 7faad6e39700 1 -- 172.21.15.115:0/3890467221 shutdown_connections 2023-07-14T18:11:19.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.592+0000 7faad6e39700 1 --2- 172.21.15.115:0/3890467221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faad0105000 0x7faad010cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.592+0000 7faad6e39700 1 --2- 172.21.15.115:0/3890467221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faad00fffb0 0x7faad0100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.592+0000 7faad6e39700 1 --2- 172.21.15.115:0/3890467221 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad0100950 0x7faad01048d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.592+0000 7faad6e39700 1 -- 172.21.15.115:0/3890467221 >> 172.21.15.115:0/3890467221 conn(0x7faad0077cd0 msgr2=0x7faad00780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:19.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.592+0000 7faad6e39700 1 -- 172.21.15.115:0/3890467221 shutdown_connections 2023-07-14T18:11:19.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.592+0000 7faad6e39700 1 -- 172.21.15.115:0/3890467221 wait complete. 2023-07-14T18:11:19.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.593+0000 7faad6e39700 1 Processor -- start 2023-07-14T18:11:19.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.593+0000 7faad6e39700 1 -- start start 2023-07-14T18:11:19.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.593+0000 7faad6e39700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad00fffb0 0x7faad0195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:19.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.593+0000 7faad6e39700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faad0100950 0x7faad01961f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:19.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.593+0000 7faad6e39700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faad0105000 0x7faad01995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:19.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.593+0000 7faad6e39700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad0199bb0 con 0x7faad00fffb0 2023-07-14T18:11:19.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.593+0000 7faad6e39700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad0199d20 con 0x7faad0105000 2023-07-14T18:11:19.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.593+0000 7faad6e39700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad0199ee0 con 0x7faad0100950 2023-07-14T18:11:19.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad4bd5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad00fffb0 0x7faad0195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:19.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad4bd5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad00fffb0 0x7faad0195cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53504/0 (socket says 172.21.15.115:53504) 2023-07-14T18:11:19.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad4bd5700 1 -- 172.21.15.115:0/1505948673 learned_addr learned my addr 172.21.15.115:0/1505948673 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:19.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faacffff700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faad0100950 0x7faad01961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:19.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad53d6700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faad0105000 0x7faad01995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:19.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad4bd5700 1 -- 172.21.15.115:0/1505948673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faad0100950 msgr2=0x7faad01961f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:19.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad4bd5700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faad0100950 0x7faad01961f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad4bd5700 1 -- 172.21.15.115:0/1505948673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faad0105000 msgr2=0x7faad01995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:19.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad4bd5700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faad0105000 0x7faad01995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad4bd5700 1 -- 172.21.15.115:0/1505948673 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc0095f0 con 0x7faad00fffb0 2023-07-14T18:11:19.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad4bd5700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad00fffb0 0x7faad0195cb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faac400c9b0 tx=0x7faac400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:19.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faacdffb700 1 -- 172.21.15.115:0/1505948673 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4007b00 con 0x7faad00fffb0 2023-07-14T18:11:19.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faacdffb700 1 -- 172.21.15.115:0/1505948673 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faac4007c60 con 0x7faad00fffb0 2023-07-14T18:11:19.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faacdffb700 1 -- 172.21.15.115:0/1505948673 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4018660 con 0x7faad00fffb0 2023-07-14T18:11:19.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad019a1c0 con 0x7faad00fffb0 2023-07-14T18:11:19.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.594+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad01a2b10 con 0x7faad00fffb0 2023-07-14T18:11:19.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.595+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad004e340 con 0x7faad00fffb0 2023-07-14T18:11:19.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.596+0000 7faacdffb700 1 -- 172.21.15.115:0/1505948673 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7faac400f450 con 0x7faad00fffb0 2023-07-14T18:11:19.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.597+0000 7faacdffb700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faac00a11e0 0x7faac00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:19.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.597+0000 7faacffff700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faac00a11e0 0x7faac00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:19.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.597+0000 7faacdffb700 1 -- 172.21.15.115:0/1505948673 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7faac40cccb0 con 0x7faad00fffb0 2023-07-14T18:11:19.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.597+0000 7faacffff700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faac00a11e0 0x7faac00a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7faabc006010 tx=0x7faabc009f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:19.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.601+0000 7faacdffb700 1 -- 172.21.15.115:0/1505948673 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faac40960b0 con 0x7faad00fffb0 2023-07-14T18:11:19.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.811+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faad01a2cd0 con 0x7faad00fffb0 2023-07-14T18:11:19.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.811+0000 7faacdffb700 1 -- 172.21.15.115:0/1505948673 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faac400fe60 con 0x7faad00fffb0 2023-07-14T18:11:19.851 INFO:teuthology.orchestra.run.smithi115.stdout:77309411388 2023-07-14T18:11:19.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faac00a11e0 msgr2=0x7faac00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:19.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faac00a11e0 0x7faac00a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7faabc006010 tx=0x7faabc009f10 comp rx=0 tx=0).stop 2023-07-14T18:11:19.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad00fffb0 msgr2=0x7faad0195cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:19.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad00fffb0 0x7faad0195cb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7faac400c9b0 tx=0x7faac400ccc0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 shutdown_connections 2023-07-14T18:11:19.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7faac00a11e0 0x7faac00a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faad0105000 0x7faad01995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7faad00fffb0 0x7faad0195cb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 --2- 172.21.15.115:0/1505948673 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faad0100950 0x7faad01961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 >> 172.21.15.115:0/1505948673 conn(0x7faad0077cd0 msgr2=0x7faad00fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:19.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 shutdown_connections 2023-07-14T18:11:19.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.813+0000 7faad6e39700 1 -- 172.21.15.115:0/1505948673 wait complete. 2023-07-14T18:11:19.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:19 smithi150 ceph-mon[128853]: pgmap v282: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T18:11:19.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.848+0000 7f7234be8700 1 -- 172.21.15.115:0/3456466499 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 msgr2=0x7f72300ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:19.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.848+0000 7f7234be8700 1 --2- 172.21.15.115:0/3456466499 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 0x7f72300ff310 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f72180099c0 tx=0x7f7218009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.848+0000 7f7234be8700 1 -- 172.21.15.115:0/3456466499 shutdown_connections 2023-07-14T18:11:19.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.848+0000 7f7234be8700 1 --2- 172.21.15.115:0/3456466499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7230101150 0x7f723010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.848+0000 7f7234be8700 1 --2- 172.21.15.115:0/3456466499 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 0x7f72300ff310 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.848+0000 7f7234be8700 1 --2- 172.21.15.115:0/3456466499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72301007a0 0x7f7230100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.848+0000 7f7234be8700 1 -- 172.21.15.115:0/3456466499 >> 172.21.15.115:0/3456466499 conn(0x7f7230075c90 msgr2=0x7f7230076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:19.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.848+0000 7f7234be8700 1 -- 172.21.15.115:0/3456466499 shutdown_connections 2023-07-14T18:11:19.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.848+0000 7f7234be8700 1 -- 172.21.15.115:0/3456466499 wait complete. 2023-07-14T18:11:19.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f7234be8700 1 Processor -- start 2023-07-14T18:11:19.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f7234be8700 1 -- start start 2023-07-14T18:11:19.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f7234be8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 0x7f7230196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:19.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f7234be8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72301007a0 0x7f7230197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:19.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f7234be8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7230101150 0x7f72301a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:19.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f7234be8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7230197a40 con 0x7f72300fef40 2023-07-14T18:11:19.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f7234be8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72301a21b0 con 0x7f7230101150 2023-07-14T18:11:19.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f7234be8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72301a2320 con 0x7f72301007a0 2023-07-14T18:11:19.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f722e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 0x7f7230196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:19.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f722e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 0x7f7230196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53518/0 (socket says 172.21.15.115:53518) 2023-07-14T18:11:19.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f722e59c700 1 -- 172.21.15.115:0/2934359297 learned_addr learned my addr 172.21.15.115:0/2934359297 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:19.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f722ed9d700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7230101150 0x7f72301a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:19.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f722e59c700 1 -- 172.21.15.115:0/2934359297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72301007a0 msgr2=0x7f7230197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:19.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f722e59c700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72301007a0 0x7f7230197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f722e59c700 1 -- 172.21.15.115:0/2934359297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7230101150 msgr2=0x7f72301a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:19.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f722e59c700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7230101150 0x7f72301a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:19.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.849+0000 7f722e59c700 1 -- 172.21.15.115:0/2934359297 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72180095f0 con 0x7f72300fef40 2023-07-14T18:11:19.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.850+0000 7f722e59c700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 0x7f7230196bf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7218009f50 tx=0x7f72180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:19.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.850+0000 7f72277fe700 1 -- 172.21.15.115:0/2934359297 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721801d070 con 0x7f72300fef40 2023-07-14T18:11:19.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.850+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72301a25a0 con 0x7f72300fef40 2023-07-14T18:11:19.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.850+0000 7f72277fe700 1 -- 172.21.15.115:0/2934359297 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f72180040b0 con 0x7f72300fef40 2023-07-14T18:11:19.894 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.850+0000 7f72277fe700 1 -- 172.21.15.115:0/2934359297 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218022cd0 con 0x7f72300fef40 2023-07-14T18:11:19.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.850+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72301a29b0 con 0x7f72300fef40 2023-07-14T18:11:19.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.851+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723004e340 con 0x7f72300fef40 2023-07-14T18:11:19.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.852+0000 7f72277fe700 1 -- 172.21.15.115:0/2934359297 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f7218022470 con 0x7f72300fef40 2023-07-14T18:11:19.895 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.852+0000 7f72277fe700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f721c0a11e0 0x7f721c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.852+0000 7f72277fe700 1 -- 172.21.15.115:0/2934359297 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f72180ce840 con 0x7f72300fef40 2023-07-14T18:11:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.852+0000 7f722dd9b700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f721c0a11e0 0x7f721c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.853+0000 7f722dd9b700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f721c0a11e0 0x7f721c0a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7220005fd0 tx=0x7f7220005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:19.896 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:19.856+0000 7f72277fe700 1 -- 172.21.15.115:0/2934359297 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7218097cd0 con 0x7f72300fef40 2023-07-14T18:11:19.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:19 smithi187 ceph-mon[129317]: pgmap v282: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T18:11:20.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.061+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f72301a2bd0 con 0x7f72300fef40 2023-07-14T18:11:20.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.062+0000 7f72277fe700 1 -- 172.21.15.115:0/2934359297 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f721809b2f0 con 0x7f72300fef40 2023-07-14T18:11:20.102 INFO:teuthology.orchestra.run.smithi115.stdout:107374182456 2023-07-14T18:11:20.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f721c0a11e0 msgr2=0x7f721c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:20.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f721c0a11e0 0x7f721c0a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7220005fd0 tx=0x7f7220005f60 comp rx=0 tx=0).stop 2023-07-14T18:11:20.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 msgr2=0x7f7230196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:20.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 0x7f7230196bf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7218009f50 tx=0x7f72180057f0 comp rx=0 tx=0).stop 2023-07-14T18:11:20.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 shutdown_connections 2023-07-14T18:11:20.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f721c0a11e0 0x7f721c0a3690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:20.104 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7230101150 0x7f72301a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:20.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f72300fef40 0x7f7230196bf0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:20.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 --2- 172.21.15.115:0/2934359297 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72301007a0 0x7f7230197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:20.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 >> 172.21.15.115:0/2934359297 conn(0x7f7230075c90 msgr2=0x7f72300fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:20.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 shutdown_connections 2023-07-14T18:11:20.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:20.064+0000 7f7234be8700 1 -- 172.21.15.115:0/2934359297 wait complete. 2023-07-14T18:11:20.466 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411385 got 77309411388 for osd.2 2023-07-14T18:11:20.467 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:20.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:20 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1505948673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T18:11:20.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:20 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2934359297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T18:11:20.735 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182453 got 107374182456 for osd.3 2023-07-14T18:11:20.735 DEBUG:teuthology.parallel:result is None 2023-07-14T18:11:20.736 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-14T18:11:20.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T18:11:20.871 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:20 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1505948673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T18:11:20.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:20 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2934359297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T18:11:20.934 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:20 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1505948673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T18:11:20.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:20 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2934359297' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T18:11:21.399 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:22.212 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:22 smithi115 ceph-mon[127451]: pgmap v283: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T18:11:22.213 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.213 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.213 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.213 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:22 smithi150 ceph-mon[128853]: pgmap v283: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T18:11:22.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:22 smithi187 ceph-mon[129317]: pgmap v283: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T18:11:22.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:22.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.732+0000 7f8da5a39700 1 -- 172.21.15.115:0/2101429544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da01008d0 msgr2=0x7f8da01048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:22.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.732+0000 7f8da5a39700 1 --2- 172.21.15.115:0/2101429544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da01008d0 0x7f8da01048a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8d880099c0 tx=0x7f8d88009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.732+0000 7f8da5a39700 1 -- 172.21.15.115:0/2101429544 shutdown_connections 2023-07-14T18:11:22.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.732+0000 7f8da5a39700 1 --2- 172.21.15.115:0/2101429544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8da0104fd0 0x7f8da010cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.732+0000 7f8da5a39700 1 --2- 172.21.15.115:0/2101429544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8da00fff30 0x7f8da0100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.732+0000 7f8da5a39700 1 --2- 172.21.15.115:0/2101429544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da01008d0 0x7f8da01048a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.732+0000 7f8da5a39700 1 -- 172.21.15.115:0/2101429544 >> 172.21.15.115:0/2101429544 conn(0x7f8da00fb790 msgr2=0x7f8da00fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:22.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.732+0000 7f8da5a39700 1 -- 172.21.15.115:0/2101429544 shutdown_connections 2023-07-14T18:11:22.774 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.732+0000 7f8da5a39700 1 -- 172.21.15.115:0/2101429544 wait complete. 2023-07-14T18:11:22.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.733+0000 7f8da5a39700 1 Processor -- start 2023-07-14T18:11:22.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.733+0000 7f8da5a39700 1 -- start start 2023-07-14T18:11:22.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.733+0000 7f8da5a39700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da00fff30 0x7f8da0195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:22.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.733+0000 7f8da5a39700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8da01008d0 0x7f8da01961f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:22.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.733+0000 7f8da5a39700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8da0104fd0 0x7f8da01995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:22.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.733+0000 7f8da5a39700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0199bb0 con 0x7f8da00fff30 2023-07-14T18:11:22.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.733+0000 7f8da5a39700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0199d20 con 0x7f8da01008d0 2023-07-14T18:11:22.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.733+0000 7f8da5a39700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0199ee0 con 0x7f8da0104fd0 2023-07-14T18:11:22.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da00fff30 0x7f8da0195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:22.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da00fff30 0x7f8da0195cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53530/0 (socket says 172.21.15.115:53530) 2023-07-14T18:11:22.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9effd700 1 -- 172.21.15.115:0/675283079 learned_addr learned my addr 172.21.15.115:0/675283079 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:22.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9f7fe700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8da0104fd0 0x7f8da01995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:22.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9e7fc700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8da01008d0 0x7f8da01961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:22.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9effd700 1 -- 172.21.15.115:0/675283079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8da0104fd0 msgr2=0x7f8da01995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:22.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9effd700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8da0104fd0 0x7f8da01995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9effd700 1 -- 172.21.15.115:0/675283079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8da01008d0 msgr2=0x7f8da01961f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:22.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9effd700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8da01008d0 0x7f8da01961f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9effd700 1 -- 172.21.15.115:0/675283079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d880095f0 con 0x7f8da00fff30 2023-07-14T18:11:22.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8d9effd700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da00fff30 0x7f8da0195cb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8d900060b0 tx=0x7f8d9000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:22.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8da4a37700 1 -- 172.21.15.115:0/675283079 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9000ddd0 con 0x7f8da00fff30 2023-07-14T18:11:22.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da019a1c0 con 0x7f8da00fff30 2023-07-14T18:11:22.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8da4a37700 1 -- 172.21.15.115:0/675283079 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8d9000f040 con 0x7f8da00fff30 2023-07-14T18:11:22.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8da4a37700 1 -- 172.21.15.115:0/675283079 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90014600 con 0x7f8da00fff30 2023-07-14T18:11:22.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.734+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da01a2b10 con 0x7f8da00fff30 2023-07-14T18:11:22.778 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.735+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da004e340 con 0x7f8da00fff30 2023-07-14T18:11:22.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.736+0000 7f8da4a37700 1 -- 172.21.15.115:0/675283079 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f8d90015400 con 0x7f8da00fff30 2023-07-14T18:11:22.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.737+0000 7f8da4a37700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8d8c0a11e0 0x7f8d8c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:22.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.737+0000 7f8d9e7fc700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8d8c0a11e0 0x7f8d8c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:22.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.737+0000 7f8da4a37700 1 -- 172.21.15.115:0/675283079 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8d900cc640 con 0x7f8da00fff30 2023-07-14T18:11:22.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.737+0000 7f8d9e7fc700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8d8c0a11e0 0x7f8d8c0a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8d88000c00 tx=0x7f8d88005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:22.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.741+0000 7f8da4a37700 1 -- 172.21.15.115:0/675283079 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8d9004f780 con 0x7f8da00fff30 2023-07-14T18:11:22.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.931+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8da00fc990 con 0x7f8d8c0a11e0 2023-07-14T18:11:22.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.943+0000 7f8da4a37700 1 -- 172.21.15.115:0/675283079 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+425967 (secure 0 0 0) 0x7f8da00fc990 con 0x7f8d8c0a11e0 2023-07-14T18:11:22.989 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:22.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.946+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8d8c0a11e0 msgr2=0x7f8d8c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:22.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.946+0000 7f8da5a39700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8d8c0a11e0 0x7f8d8c0a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8d88000c00 tx=0x7f8d88005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.946+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da00fff30 msgr2=0x7f8da0195cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:22.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.946+0000 7f8da5a39700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da00fff30 0x7f8da0195cb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8d900060b0 tx=0x7f8d9000d3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.947+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 shutdown_connections 2023-07-14T18:11:22.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.947+0000 7f8da5a39700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f8d8c0a11e0 0x7f8d8c0a3690 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.947+0000 7f8da5a39700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8da0104fd0 0x7f8da01995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.947+0000 7f8da5a39700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8da00fff30 0x7f8da0195cb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.947+0000 7f8da5a39700 1 --2- 172.21.15.115:0/675283079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8da01008d0 0x7f8da01961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:22.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.947+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 >> 172.21.15.115:0/675283079 conn(0x7f8da00fb790 msgr2=0x7f8da00fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:22.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.947+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 shutdown_connections 2023-07-14T18:11:22.996 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:22.947+0000 7f8da5a39700 1 -- 172.21.15.115:0/675283079 wait complete. 2023-07-14T18:11:22.996 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-07-14T18:11:23.468 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2023-07-14T18:11:22.897015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653459,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":416,"num_read_kb":318,"num_write":337,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337592,"kb_used_data":13400,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604232,"statfs":{"total":1151940427776,"available":1151594733568,"internally_reserved":0,"allocated":13721600,"data_stored":7581014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":86,"num_read_kb":42,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003590"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089430+0000","last_change":"2023-07-14T18:10:18.110819+0000","last_active":"2023-07-14T18:10:24.089430+0000","last_peered":"2023-07-14T18:10:24.089430+0000","last_clean":"2023-07-14T18:10:24.089430+0000","last_became_active":"2023-07-14T18:10:18.110180+0000","last_became_peered":"2023-07-14T18:10:18.110180+0000","last_unstale":"2023-07-14T18:10:24.089430+0000","last_undegraded":"2023-07-14T18:10:24.089430+0000","last_fullsized":"2023-07-14T18:10:24.089430+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:08:50.579212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440800+0000","last_change":"2023-07-14T18:10:18.117539+0000","last_active":"2023-07-14T18:10:23.440800+0000","last_peered":"2023-07-14T18:10:23.440800+0000","last_clean":"2023-07-14T18:10:23.440800+0000","last_became_active":"2023-07-14T18:10:18.115956+0000","last_became_peered":"2023-07-14T18:10:18.115956+0000","last_unstale":"2023-07-14T18:10:23.440800+0000","last_undegraded":"2023-07-14T18:10:23.440800+0000","last_fullsized":"2023-07-14T18:10:23.440800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:22.723383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440427+0000","last_change":"2023-07-14T18:10:18.120582+0000","last_active":"2023-07-14T18:10:23.440427+0000","last_peered":"2023-07-14T18:10:23.440427+0000","last_clean":"2023-07-14T18:10:23.440427+0000","last_became_active":"2023-07-14T18:10:18.120296+0000","last_became_peered":"2023-07-14T18:10:18.120296+0000","last_unstale":"2023-07-14T18:10:23.440427+0000","last_undegraded":"2023-07-14T18:10:23.440427+0000","last_fullsized":"2023-07-14T18:10:23.440427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:54.381627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438612+0000","last_change":"2023-07-14T18:10:18.112250+0000","last_active":"2023-07-14T18:10:23.438612+0000","last_peered":"2023-07-14T18:10:23.438612+0000","last_clean":"2023-07-14T18:10:23.438612+0000","last_became_active":"2023-07-14T18:10:18.111514+0000","last_became_peered":"2023-07-14T18:10:18.111514+0000","last_unstale":"2023-07-14T18:10:23.438612+0000","last_undegraded":"2023-07-14T18:10:23.438612+0000","last_fullsized":"2023-07-14T18:10:23.438612+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:02:30.280559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438548+0000","last_change":"2023-07-14T18:10:18.111127+0000","last_active":"2023-07-14T18:10:23.438548+0000","last_peered":"2023-07-14T18:10:23.438548+0000","last_clean":"2023-07-14T18:10:23.438548+0000","last_became_active":"2023-07-14T18:10:18.110816+0000","last_became_peered":"2023-07-14T18:10:18.110816+0000","last_unstale":"2023-07-14T18:10:23.438548+0000","last_undegraded":"2023-07-14T18:10:23.438548+0000","last_fullsized":"2023-07-14T18:10:23.438548+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:05:59.027134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734705+0000","last_change":"2023-07-14T18:10:18.113436+0000","last_active":"2023-07-14T18:10:23.734705+0000","last_peered":"2023-07-14T18:10:23.734705+0000","last_clean":"2023-07-14T18:10:23.734705+0000","last_became_active":"2023-07-14T18:10:18.113172+0000","last_became_peered":"2023-07-14T18:10:18.113172+0000","last_unstale":"2023-07-14T18:10:23.734705+0000","last_undegraded":"2023-07-14T18:10:23.734705+0000","last_fullsized":"2023-07-14T18:10:23.734705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:33:57.446784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089034+0000","last_change":"2023-07-14T18:10:18.116720+0000","last_active":"2023-07-14T18:10:24.089034+0000","last_peered":"2023-07-14T18:10:24.089034+0000","last_clean":"2023-07-14T18:10:24.089034+0000","last_became_active":"2023-07-14T18:10:18.116361+0000","last_became_peered":"2023-07-14T18:10:18.116361+0000","last_unstale":"2023-07-14T18:10:24.089034+0000","last_undegraded":"2023-07-14T18:10:24.089034+0000","last_fullsized":"2023-07-14T18:10:24.089034+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:54.195458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088667+0000","last_change":"2023-07-14T18:10:18.111981+0000","last_active":"2023-07-14T18:10:24.088667+0000","last_peered":"2023-07-14T18:10:24.088667+0000","last_clean":"2023-07-14T18:10:24.088667+0000","last_became_active":"2023-07-14T18:10:18.111666+0000","last_became_peered":"2023-07-14T18:10:18.111666+0000","last_unstale":"2023-07-14T18:10:24.088667+0000","last_undegraded":"2023-07-14T18:10:24.088667+0000","last_fullsized":"2023-07-14T18:10:24.088667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:07:35.710429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089557+0000","last_change":"2023-07-14T18:10:18.119497+0000","last_active":"2023-07-14T18:10:24.089557+0000","last_peered":"2023-07-14T18:10:24.089557+0000","last_clean":"2023-07-14T18:10:24.089557+0000","last_became_active":"2023-07-14T18:10:18.119090+0000","last_became_peered":"2023-07-14T18:10:18.119090+0000","last_unstale":"2023-07-14T18:10:24.089557+0000","last_undegraded":"2023-07-14T18:10:24.089557+0000","last_fullsized":"2023-07-14T18:10:24.089557+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:15:13.934598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.735011+0000","last_change":"2023-07-14T18:10:18.107353+0000","last_active":"2023-07-14T18:10:23.735011+0000","last_peered":"2023-07-14T18:10:23.735011+0000","last_clean":"2023-07-14T18:10:23.735011+0000","last_became_active":"2023-07-14T18:10:18.106933+0000","last_became_peered":"2023-07-14T18:10:18.106933+0000","last_unstale":"2023-07-14T18:10:23.735011+0000","last_undegraded":"2023-07-14T18:10:23.735011+0000","last_fullsized":"2023-07-14T18:10:23.735011+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:20:25.508991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089317+0000","last_change":"2023-07-14T18:10:18.121064+0000","last_active":"2023-07-14T18:10:24.089317+0000","last_peered":"2023-07-14T18:10:24.089317+0000","last_clean":"2023-07-14T18:10:24.089317+0000","last_became_active":"2023-07-14T18:10:18.120758+0000","last_became_peered":"2023-07-14T18:10:18.120758+0000","last_unstale":"2023-07-14T18:10:24.089317+0000","last_undegraded":"2023-07-14T18:10:24.089317+0000","last_fullsized":"2023-07-14T18:10:24.089317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:42.972896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438568+0000","last_change":"2023-07-14T18:10:18.112436+0000","last_active":"2023-07-14T18:10:23.438568+0000","last_peered":"2023-07-14T18:10:23.438568+0000","last_clean":"2023-07-14T18:10:23.438568+0000","last_became_active":"2023-07-14T18:10:18.111942+0000","last_became_peered":"2023-07-14T18:10:18.111942+0000","last_unstale":"2023-07-14T18:10:23.438568+0000","last_undegraded":"2023-07-14T18:10:23.438568+0000","last_fullsized":"2023-07-14T18:10:23.438568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:57.856511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845884+0000","last_change":"2023-07-14T18:10:18.106134+0000","last_active":"2023-07-14T18:10:23.845884+0000","last_peered":"2023-07-14T18:10:23.845884+0000","last_clean":"2023-07-14T18:10:23.845884+0000","last_became_active":"2023-07-14T18:10:18.105481+0000","last_became_peered":"2023-07-14T18:10:18.105481+0000","last_unstale":"2023-07-14T18:10:23.845884+0000","last_undegraded":"2023-07-14T18:10:23.845884+0000","last_fullsized":"2023-07-14T18:10:23.845884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:53.958762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440197+0000","last_change":"2023-07-14T18:10:18.117399+0000","last_active":"2023-07-14T18:10:23.440197+0000","last_peered":"2023-07-14T18:10:23.440197+0000","last_clean":"2023-07-14T18:10:23.440197+0000","last_became_active":"2023-07-14T18:10:18.117094+0000","last_became_peered":"2023-07-14T18:10:18.117094+0000","last_unstale":"2023-07-14T18:10:23.440197+0000","last_undegraded":"2023-07-14T18:10:23.440197+0000","last_fullsized":"2023-07-14T18:10:23.440197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:10:09.445305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440157+0000","last_change":"2023-07-14T18:10:18.117150+0000","last_active":"2023-07-14T18:10:23.440157+0000","last_peered":"2023-07-14T18:10:23.440157+0000","last_clean":"2023-07-14T18:10:23.440157+0000","last_became_active":"2023-07-14T18:10:18.116805+0000","last_became_peered":"2023-07-14T18:10:18.116805+0000","last_unstale":"2023-07-14T18:10:23.440157+0000","last_undegraded":"2023-07-14T18:10:23.440157+0000","last_fullsized":"2023-07-14T18:10:23.440157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:48:43.803465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088549+0000","last_change":"2023-07-14T18:10:18.120868+0000","last_active":"2023-07-14T18:10:24.088549+0000","last_peered":"2023-07-14T18:10:24.088549+0000","last_clean":"2023-07-14T18:10:24.088549+0000","last_became_active":"2023-07-14T18:10:18.120083+0000","last_became_peered":"2023-07-14T18:10:18.120083+0000","last_unstale":"2023-07-14T18:10:24.088549+0000","last_undegraded":"2023-07-14T18:10:24.088549+0000","last_fullsized":"2023-07-14T18:10:24.088549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:50:07.162775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089298+0000","last_change":"2023-07-14T18:10:18.111011+0000","last_active":"2023-07-14T18:10:24.089298+0000","last_peered":"2023-07-14T18:10:24.089298+0000","last_clean":"2023-07-14T18:10:24.089298+0000","last_became_active":"2023-07-14T18:10:18.110517+0000","last_became_peered":"2023-07-14T18:10:18.110517+0000","last_unstale":"2023-07-14T18:10:24.089298+0000","last_undegraded":"2023-07-14T18:10:24.089298+0000","last_fullsized":"2023-07-14T18:10:24.089298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:00:45.278711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089798+0000","last_change":"2023-07-14T18:10:18.110739+0000","last_active":"2023-07-14T18:10:24.089798+0000","last_peered":"2023-07-14T18:10:24.089798+0000","last_clean":"2023-07-14T18:10:24.089798+0000","last_became_active":"2023-07-14T18:10:18.110442+0000","last_became_peered":"2023-07-14T18:10:18.110442+0000","last_unstale":"2023-07-14T18:10:24.089798+0000","last_undegraded":"2023-07-14T18:10:24.089798+0000","last_fullsized":"2023-07-14T18:10:24.089798+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:45.130540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550066+0000","last_change":"2023-07-14T18:10:18.109955+0000","last_active":"2023-07-14T18:10:23.550066+0000","last_peered":"2023-07-14T18:10:23.550066+0000","last_clean":"2023-07-14T18:10:23.550066+0000","last_became_active":"2023-07-14T18:10:18.109651+0000","last_became_peered":"2023-07-14T18:10:18.109651+0000","last_unstale":"2023-07-14T18:10:23.550066+0000","last_undegraded":"2023-07-14T18:10:23.550066+0000","last_fullsized":"2023-07-14T18:10:23.550066+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:22:46.479116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549977+0000","last_change":"2023-07-14T18:10:22.457996+0000","last_active":"2023-07-14T18:10:23.549977+0000","last_peered":"2023-07-14T18:10:23.549977+0000","last_clean":"2023-07-14T18:10:23.549977+0000","last_became_active":"2023-07-14T18:10:22.457573+0000","last_became_peered":"2023-07-14T18:10:22.457573+0000","last_unstale":"2023-07-14T18:10:23.549977+0000","last_undegraded":"2023-07-14T18:10:23.549977+0000","last_fullsized":"2023-07-14T18:10:23.549977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:17.148116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429932+0000","last_change":"2023-07-14T18:10:17.216780+0000","last_active":"2023-07-14T18:10:37.429932+0000","last_peered":"2023-07-14T18:10:37.429932+0000","last_clean":"2023-07-14T18:10:37.429932+0000","last_became_active":"2023-07-14T18:10:17.216423+0000","last_became_peered":"2023-07-14T18:10:17.216423+0000","last_unstale":"2023-07-14T18:10:37.429932+0000","last_undegraded":"2023-07-14T18:10:37.429932+0000","last_fullsized":"2023-07-14T18:10:37.429932+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:24.522131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089127+0000","last_change":"2023-07-14T18:10:18.108615+0000","last_active":"2023-07-14T18:10:24.089127+0000","last_peered":"2023-07-14T18:10:24.089127+0000","last_clean":"2023-07-14T18:10:24.089127+0000","last_became_active":"2023-07-14T18:10:18.108316+0000","last_became_peered":"2023-07-14T18:10:18.108316+0000","last_unstale":"2023-07-14T18:10:24.089127+0000","last_undegraded":"2023-07-14T18:10:24.089127+0000","last_fullsized":"2023-07-14T18:10:24.089127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:27:25.084342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438203+0000","last_change":"2023-07-14T18:10:22.453040+0000","last_active":"2023-07-14T18:10:23.438203+0000","last_peered":"2023-07-14T18:10:23.438203+0000","last_clean":"2023-07-14T18:10:23.438203+0000","last_became_active":"2023-07-14T18:10:22.452643+0000","last_became_peered":"2023-07-14T18:10:22.452643+0000","last_unstale":"2023-07-14T18:10:23.438203+0000","last_undegraded":"2023-07-14T18:10:23.438203+0000","last_fullsized":"2023-07-14T18:10:23.438203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:18.191433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089445+0000","last_change":"2023-07-14T18:10:17.214085+0000","last_active":"2023-07-14T18:10:24.089445+0000","last_peered":"2023-07-14T18:10:24.089445+0000","last_clean":"2023-07-14T18:10:24.089445+0000","last_became_active":"2023-07-14T18:10:17.213772+0000","last_became_peered":"2023-07-14T18:10:17.213772+0000","last_unstale":"2023-07-14T18:10:24.089445+0000","last_undegraded":"2023-07-14T18:10:24.089445+0000","last_fullsized":"2023-07-14T18:10:24.089445+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:16:35.586284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440705+0000","last_change":"2023-07-14T18:10:18.120532+0000","last_active":"2023-07-14T18:10:23.440705+0000","last_peered":"2023-07-14T18:10:23.440705+0000","last_clean":"2023-07-14T18:10:23.440705+0000","last_became_active":"2023-07-14T18:10:18.120199+0000","last_became_peered":"2023-07-14T18:10:18.120199+0000","last_unstale":"2023-07-14T18:10:23.440705+0000","last_undegraded":"2023-07-14T18:10:23.440705+0000","last_fullsized":"2023-07-14T18:10:23.440705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:05:38.292570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438439+0000","last_change":"2023-07-14T18:10:22.458820+0000","last_active":"2023-07-14T18:10:23.438439+0000","last_peered":"2023-07-14T18:10:23.438439+0000","last_clean":"2023-07-14T18:10:23.438439+0000","last_became_active":"2023-07-14T18:10:22.458480+0000","last_became_peered":"2023-07-14T18:10:22.458480+0000","last_unstale":"2023-07-14T18:10:23.438439+0000","last_undegraded":"2023-07-14T18:10:23.438439+0000","last_fullsized":"2023-07-14T18:10:23.438439+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:45.901536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428015+0000","last_change":"2023-07-14T18:10:17.212329+0000","last_active":"2023-07-14T18:10:37.428015+0000","last_peered":"2023-07-14T18:10:37.428015+0000","last_clean":"2023-07-14T18:10:37.428015+0000","last_became_active":"2023-07-14T18:10:17.212013+0000","last_became_peered":"2023-07-14T18:10:17.212013+0000","last_unstale":"2023-07-14T18:10:37.428015+0000","last_undegraded":"2023-07-14T18:10:37.428015+0000","last_fullsized":"2023-07-14T18:10:37.428015+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:19:15.950647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089229+0000","last_change":"2023-07-14T18:10:18.114621+0000","last_active":"2023-07-14T18:10:24.089229+0000","last_peered":"2023-07-14T18:10:24.089229+0000","last_clean":"2023-07-14T18:10:24.089229+0000","last_became_active":"2023-07-14T18:10:18.114330+0000","last_became_peered":"2023-07-14T18:10:18.114330+0000","last_unstale":"2023-07-14T18:10:24.089229+0000","last_undegraded":"2023-07-14T18:10:24.089229+0000","last_fullsized":"2023-07-14T18:10:24.089229+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:47:21.060539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438244+0000","last_change":"2023-07-14T18:10:22.453179+0000","last_active":"2023-07-14T18:10:23.438244+0000","last_peered":"2023-07-14T18:10:23.438244+0000","last_clean":"2023-07-14T18:10:23.438244+0000","last_became_active":"2023-07-14T18:10:22.452745+0000","last_became_peered":"2023-07-14T18:10:22.452745+0000","last_unstale":"2023-07-14T18:10:23.438244+0000","last_undegraded":"2023-07-14T18:10:23.438244+0000","last_fullsized":"2023-07-14T18:10:23.438244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:22:16.784956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438868+0000","last_change":"2023-07-14T18:10:17.215115+0000","last_active":"2023-07-14T18:10:23.438868+0000","last_peered":"2023-07-14T18:10:23.438868+0000","last_clean":"2023-07-14T18:10:23.438868+0000","last_became_active":"2023-07-14T18:10:17.214813+0000","last_became_peered":"2023-07-14T18:10:17.214813+0000","last_unstale":"2023-07-14T18:10:23.438868+0000","last_undegraded":"2023-07-14T18:10:23.438868+0000","last_fullsized":"2023-07-14T18:10:23.438868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:48:32.267436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090280+0000","last_change":"2023-07-14T18:10:18.119987+0000","last_active":"2023-07-14T18:10:24.090280+0000","last_peered":"2023-07-14T18:10:24.090280+0000","last_clean":"2023-07-14T18:10:24.090280+0000","last_became_active":"2023-07-14T18:10:18.119695+0000","last_became_peered":"2023-07-14T18:10:18.119695+0000","last_unstale":"2023-07-14T18:10:24.090280+0000","last_undegraded":"2023-07-14T18:10:24.090280+0000","last_fullsized":"2023-07-14T18:10:24.090280+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:46.295719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550007+0000","last_change":"2023-07-14T18:10:22.449689+0000","last_active":"2023-07-14T18:10:23.550007+0000","last_peered":"2023-07-14T18:10:23.550007+0000","last_clean":"2023-07-14T18:10:23.550007+0000","last_became_active":"2023-07-14T18:10:22.449273+0000","last_became_peered":"2023-07-14T18:10:22.449273+0000","last_unstale":"2023-07-14T18:10:23.550007+0000","last_undegraded":"2023-07-14T18:10:23.550007+0000","last_fullsized":"2023-07-14T18:10:23.550007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:54:23.642287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:20.487923+0000","last_change":"2023-07-14T18:10:17.216438+0000","last_active":"2023-07-14T18:11:20.487923+0000","last_peered":"2023-07-14T18:11:20.487923+0000","last_clean":"2023-07-14T18:11:20.487923+0000","last_became_active":"2023-07-14T18:10:17.216169+0000","last_became_peered":"2023-07-14T18:10:17.216169+0000","last_unstale":"2023-07-14T18:11:20.487923+0000","last_undegraded":"2023-07-14T18:11:20.487923+0000","last_fullsized":"2023-07-14T18:11:20.487923+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:39.977001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438762+0000","last_change":"2023-07-14T18:10:18.112917+0000","last_active":"2023-07-14T18:10:23.438762+0000","last_peered":"2023-07-14T18:10:23.438762+0000","last_clean":"2023-07-14T18:10:23.438762+0000","last_became_active":"2023-07-14T18:10:18.112631+0000","last_became_peered":"2023-07-14T18:10:18.112631+0000","last_unstale":"2023-07-14T18:10:23.438762+0000","last_undegraded":"2023-07-14T18:10:23.438762+0000","last_fullsized":"2023-07-14T18:10:23.438762+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:14:01.774309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549936+0000","last_change":"2023-07-14T18:10:22.454278+0000","last_active":"2023-07-14T18:10:23.549936+0000","last_peered":"2023-07-14T18:10:23.549936+0000","last_clean":"2023-07-14T18:10:23.549936+0000","last_became_active":"2023-07-14T18:10:22.453834+0000","last_became_peered":"2023-07-14T18:10:22.453834+0000","last_unstale":"2023-07-14T18:10:23.549936+0000","last_undegraded":"2023-07-14T18:10:23.549936+0000","last_fullsized":"2023-07-14T18:10:23.549936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:31:24.983615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430580+0000","last_change":"2023-07-14T18:10:17.210217+0000","last_active":"2023-07-14T18:10:37.430580+0000","last_peered":"2023-07-14T18:10:37.430580+0000","last_clean":"2023-07-14T18:10:37.430580+0000","last_became_active":"2023-07-14T18:10:17.209953+0000","last_became_peered":"2023-07-14T18:10:17.209953+0000","last_unstale":"2023-07-14T18:10:37.430580+0000","last_undegraded":"2023-07-14T18:10:37.430580+0000","last_fullsized":"2023-07-14T18:10:37.430580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:19:46.091265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090402+0000","last_change":"2023-07-14T18:10:18.113009+0000","last_active":"2023-07-14T18:10:24.090402+0000","last_peered":"2023-07-14T18:10:24.090402+0000","last_clean":"2023-07-14T18:10:24.090402+0000","last_became_active":"2023-07-14T18:10:18.112695+0000","last_became_peered":"2023-07-14T18:10:18.112695+0000","last_unstale":"2023-07-14T18:10:24.090402+0000","last_undegraded":"2023-07-14T18:10:24.090402+0000","last_fullsized":"2023-07-14T18:10:24.090402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:06:04.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089159+0000","last_change":"2023-07-14T18:10:22.457687+0000","last_active":"2023-07-14T18:10:24.089159+0000","last_peered":"2023-07-14T18:10:24.089159+0000","last_clean":"2023-07-14T18:10:24.089159+0000","last_became_active":"2023-07-14T18:10:22.457270+0000","last_became_peered":"2023-07-14T18:10:22.457270+0000","last_unstale":"2023-07-14T18:10:24.089159+0000","last_undegraded":"2023-07-14T18:10:24.089159+0000","last_fullsized":"2023-07-14T18:10:24.089159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:37:58.192527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429469+0000","last_change":"2023-07-14T18:10:17.209638+0000","last_active":"2023-07-14T18:10:37.429469+0000","last_peered":"2023-07-14T18:10:37.429469+0000","last_clean":"2023-07-14T18:10:37.429469+0000","last_became_active":"2023-07-14T18:10:17.209409+0000","last_became_peered":"2023-07-14T18:10:17.209409+0000","last_unstale":"2023-07-14T18:10:37.429469+0000","last_undegraded":"2023-07-14T18:10:37.429469+0000","last_fullsized":"2023-07-14T18:10:37.429469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:32.419102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088533+0000","last_change":"2023-07-14T18:10:18.109479+0000","last_active":"2023-07-14T18:10:24.088533+0000","last_peered":"2023-07-14T18:10:24.088533+0000","last_clean":"2023-07-14T18:10:24.088533+0000","last_became_active":"2023-07-14T18:10:18.109161+0000","last_became_peered":"2023-07-14T18:10:18.109161+0000","last_unstale":"2023-07-14T18:10:24.088533+0000","last_undegraded":"2023-07-14T18:10:24.088533+0000","last_fullsized":"2023-07-14T18:10:24.088533+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:05.204295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440596+0000","last_change":"2023-07-14T18:10:22.453731+0000","last_active":"2023-07-14T18:10:23.440596+0000","last_peered":"2023-07-14T18:10:23.440596+0000","last_clean":"2023-07-14T18:10:23.440596+0000","last_became_active":"2023-07-14T18:10:22.453311+0000","last_became_peered":"2023-07-14T18:10:22.453311+0000","last_unstale":"2023-07-14T18:10:23.440596+0000","last_undegraded":"2023-07-14T18:10:23.440596+0000","last_fullsized":"2023-07-14T18:10:23.440596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:35.892698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089898+0000","last_change":"2023-07-14T18:10:17.218050+0000","last_active":"2023-07-14T18:10:24.089898+0000","last_peered":"2023-07-14T18:10:24.089898+0000","last_clean":"2023-07-14T18:10:24.089898+0000","last_became_active":"2023-07-14T18:10:17.217793+0000","last_became_peered":"2023-07-14T18:10:17.217793+0000","last_unstale":"2023-07-14T18:10:24.089898+0000","last_undegraded":"2023-07-14T18:10:24.089898+0000","last_fullsized":"2023-07-14T18:10:24.089898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:23.781098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089467+0000","last_change":"2023-07-14T18:10:18.111598+0000","last_active":"2023-07-14T18:10:24.089467+0000","last_peered":"2023-07-14T18:10:24.089467+0000","last_clean":"2023-07-14T18:10:24.089467+0000","last_became_active":"2023-07-14T18:10:18.111244+0000","last_became_peered":"2023-07-14T18:10:18.111244+0000","last_unstale":"2023-07-14T18:10:24.089467+0000","last_undegraded":"2023-07-14T18:10:24.089467+0000","last_fullsized":"2023-07-14T18:10:24.089467+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:26:47.187587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088793+0000","last_change":"2023-07-14T18:10:22.447676+0000","last_active":"2023-07-14T18:10:24.088793+0000","last_peered":"2023-07-14T18:10:24.088793+0000","last_clean":"2023-07-14T18:10:24.088793+0000","last_became_active":"2023-07-14T18:10:22.446648+0000","last_became_peered":"2023-07-14T18:10:22.446648+0000","last_unstale":"2023-07-14T18:10:24.088793+0000","last_undegraded":"2023-07-14T18:10:24.088793+0000","last_fullsized":"2023-07-14T18:10:24.088793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:17:30.941899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:19.486536+0000","last_change":"2023-07-14T18:10:17.213897+0000","last_active":"2023-07-14T18:11:19.486536+0000","last_peered":"2023-07-14T18:11:19.486536+0000","last_clean":"2023-07-14T18:11:19.486536+0000","last_became_active":"2023-07-14T18:10:17.213547+0000","last_became_peered":"2023-07-14T18:10:17.213547+0000","last_unstale":"2023-07-14T18:11:19.486536+0000","last_undegraded":"2023-07-14T18:11:19.486536+0000","last_fullsized":"2023-07-14T18:11:19.486536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:44.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089327+0000","last_change":"2023-07-14T18:10:18.114776+0000","last_active":"2023-07-14T18:10:24.089327+0000","last_peered":"2023-07-14T18:10:24.089327+0000","last_clean":"2023-07-14T18:10:24.089327+0000","last_became_active":"2023-07-14T18:10:18.112469+0000","last_became_peered":"2023-07-14T18:10:18.112469+0000","last_unstale":"2023-07-14T18:10:24.089327+0000","last_undegraded":"2023-07-14T18:10:24.089327+0000","last_fullsized":"2023-07-14T18:10:24.089327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:30.864496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088873+0000","last_change":"2023-07-14T18:10:22.454215+0000","last_active":"2023-07-14T18:10:24.088873+0000","last_peered":"2023-07-14T18:10:24.088873+0000","last_clean":"2023-07-14T18:10:24.088873+0000","last_became_active":"2023-07-14T18:10:22.453889+0000","last_became_peered":"2023-07-14T18:10:22.453889+0000","last_unstale":"2023-07-14T18:10:24.088873+0000","last_undegraded":"2023-07-14T18:10:24.088873+0000","last_fullsized":"2023-07-14T18:10:24.088873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:30:09.159482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981368+0000","last_change":"2023-07-14T18:10:17.212602+0000","last_active":"2023-07-14T18:10:36.981368+0000","last_peered":"2023-07-14T18:10:36.981368+0000","last_clean":"2023-07-14T18:10:36.981368+0000","last_became_active":"2023-07-14T18:10:17.212087+0000","last_became_peered":"2023-07-14T18:10:17.212087+0000","last_unstale":"2023-07-14T18:10:36.981368+0000","last_undegraded":"2023-07-14T18:10:36.981368+0000","last_fullsized":"2023-07-14T18:10:36.981368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:30:00.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440688+0000","last_change":"2023-07-14T18:10:18.116972+0000","last_active":"2023-07-14T18:10:23.440688+0000","last_peered":"2023-07-14T18:10:23.440688+0000","last_clean":"2023-07-14T18:10:23.440688+0000","last_became_active":"2023-07-14T18:10:18.116423+0000","last_became_peered":"2023-07-14T18:10:18.116423+0000","last_unstale":"2023-07-14T18:10:23.440688+0000","last_undegraded":"2023-07-14T18:10:23.440688+0000","last_fullsized":"2023-07-14T18:10:23.440688+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:53.315535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438563+0000","last_change":"2023-07-14T18:10:22.450207+0000","last_active":"2023-07-14T18:10:23.438563+0000","last_peered":"2023-07-14T18:10:23.438563+0000","last_clean":"2023-07-14T18:10:23.438563+0000","last_became_active":"2023-07-14T18:10:22.449826+0000","last_became_peered":"2023-07-14T18:10:22.449826+0000","last_unstale":"2023-07-14T18:10:23.438563+0000","last_undegraded":"2023-07-14T18:10:23.438563+0000","last_fullsized":"2023-07-14T18:10:23.438563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:18.013945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.982272+0000","last_change":"2023-07-14T18:10:17.213769+0000","last_active":"2023-07-14T18:10:36.982272+0000","last_peered":"2023-07-14T18:10:36.982272+0000","last_clean":"2023-07-14T18:10:36.982272+0000","last_became_active":"2023-07-14T18:10:17.213465+0000","last_became_peered":"2023-07-14T18:10:17.213465+0000","last_unstale":"2023-07-14T18:10:36.982272+0000","last_undegraded":"2023-07-14T18:10:36.982272+0000","last_fullsized":"2023-07-14T18:10:36.982272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:38:07.890716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089153+0000","last_change":"2023-07-14T18:10:18.122764+0000","last_active":"2023-07-14T18:10:24.089153+0000","last_peered":"2023-07-14T18:10:24.089153+0000","last_clean":"2023-07-14T18:10:24.089153+0000","last_became_active":"2023-07-14T18:10:18.120522+0000","last_became_peered":"2023-07-14T18:10:18.120522+0000","last_unstale":"2023-07-14T18:10:24.089153+0000","last_undegraded":"2023-07-14T18:10:24.089153+0000","last_fullsized":"2023-07-14T18:10:24.089153+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:54:07.567008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089725+0000","last_change":"2023-07-14T18:10:22.455635+0000","last_active":"2023-07-14T18:10:24.089725+0000","last_peered":"2023-07-14T18:10:24.089725+0000","last_clean":"2023-07-14T18:10:24.089725+0000","last_became_active":"2023-07-14T18:10:22.455128+0000","last_became_peered":"2023-07-14T18:10:22.455128+0000","last_unstale":"2023-07-14T18:10:24.089725+0000","last_undegraded":"2023-07-14T18:10:24.089725+0000","last_fullsized":"2023-07-14T18:10:24.089725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:01.364723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":149,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:20.487400+0000","last_change":"2023-07-14T18:10:17.215255+0000","last_active":"2023-07-14T18:11:20.487400+0000","last_peered":"2023-07-14T18:11:20.487400+0000","last_clean":"2023-07-14T18:11:20.487400+0000","last_became_active":"2023-07-14T18:10:17.214900+0000","last_became_peered":"2023-07-14T18:10:17.214900+0000","last_unstale":"2023-07-14T18:11:20.487400+0000","last_undegraded":"2023-07-14T18:11:20.487400+0000","last_fullsized":"2023-07-14T18:11:20.487400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:01.480644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":96,"num_read_kb":104,"num_write":32,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438902+0000","last_change":"2023-07-14T18:10:18.115355+0000","last_active":"2023-07-14T18:10:23.438902+0000","last_peered":"2023-07-14T18:10:23.438902+0000","last_clean":"2023-07-14T18:10:23.438902+0000","last_became_active":"2023-07-14T18:10:18.115023+0000","last_became_peered":"2023-07-14T18:10:18.115023+0000","last_unstale":"2023-07-14T18:10:23.438902+0000","last_undegraded":"2023-07-14T18:10:23.438902+0000","last_fullsized":"2023-07-14T18:10:23.438902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:16.057123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090803+0000","last_change":"2023-07-14T18:10:22.453113+0000","last_active":"2023-07-14T18:10:24.090803+0000","last_peered":"2023-07-14T18:10:24.090803+0000","last_clean":"2023-07-14T18:10:24.090803+0000","last_became_active":"2023-07-14T18:10:22.452688+0000","last_became_peered":"2023-07-14T18:10:22.452688+0000","last_unstale":"2023-07-14T18:10:24.090803+0000","last_undegraded":"2023-07-14T18:10:24.090803+0000","last_fullsized":"2023-07-14T18:10:24.090803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:47.829022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430020+0000","last_change":"2023-07-14T18:10:17.210924+0000","last_active":"2023-07-14T18:10:37.430020+0000","last_peered":"2023-07-14T18:10:37.430020+0000","last_clean":"2023-07-14T18:10:37.430020+0000","last_became_active":"2023-07-14T18:10:17.210604+0000","last_became_peered":"2023-07-14T18:10:17.210604+0000","last_unstale":"2023-07-14T18:10:37.430020+0000","last_undegraded":"2023-07-14T18:10:37.430020+0000","last_fullsized":"2023-07-14T18:10:37.430020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:10:28.635558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845862+0000","last_change":"2023-07-14T18:10:18.115094+0000","last_active":"2023-07-14T18:10:23.845862+0000","last_peered":"2023-07-14T18:10:23.845862+0000","last_clean":"2023-07-14T18:10:23.845862+0000","last_became_active":"2023-07-14T18:10:18.114538+0000","last_became_peered":"2023-07-14T18:10:18.114538+0000","last_unstale":"2023-07-14T18:10:23.845862+0000","last_undegraded":"2023-07-14T18:10:23.845862+0000","last_fullsized":"2023-07-14T18:10:23.845862+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:19:15.916223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088869+0000","last_change":"2023-07-14T18:10:22.459151+0000","last_active":"2023-07-14T18:10:24.088869+0000","last_peered":"2023-07-14T18:10:24.088869+0000","last_clean":"2023-07-14T18:10:24.088869+0000","last_became_active":"2023-07-14T18:10:22.458550+0000","last_became_peered":"2023-07-14T18:10:22.458550+0000","last_unstale":"2023-07-14T18:10:24.088869+0000","last_undegraded":"2023-07-14T18:10:24.088869+0000","last_fullsized":"2023-07-14T18:10:24.088869+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:11.750342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.982150+0000","last_change":"2023-07-14T18:10:17.216475+0000","last_active":"2023-07-14T18:10:36.982150+0000","last_peered":"2023-07-14T18:10:36.982150+0000","last_clean":"2023-07-14T18:10:36.982150+0000","last_became_active":"2023-07-14T18:10:17.216148+0000","last_became_peered":"2023-07-14T18:10:17.216148+0000","last_unstale":"2023-07-14T18:10:36.982150+0000","last_undegraded":"2023-07-14T18:10:36.982150+0000","last_fullsized":"2023-07-14T18:10:36.982150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:00:50.106730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089545+0000","last_change":"2023-07-14T18:10:18.121764+0000","last_active":"2023-07-14T18:10:24.089545+0000","last_peered":"2023-07-14T18:10:24.089545+0000","last_clean":"2023-07-14T18:10:24.089545+0000","last_became_active":"2023-07-14T18:10:18.121458+0000","last_became_peered":"2023-07-14T18:10:18.121458+0000","last_unstale":"2023-07-14T18:10:24.089545+0000","last_undegraded":"2023-07-14T18:10:24.089545+0000","last_fullsized":"2023-07-14T18:10:24.089545+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:26:22.145994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089867+0000","last_change":"2023-07-14T18:10:22.457737+0000","last_active":"2023-07-14T18:10:24.089867+0000","last_peered":"2023-07-14T18:10:24.089867+0000","last_clean":"2023-07-14T18:10:24.089867+0000","last_became_active":"2023-07-14T18:10:22.457359+0000","last_became_peered":"2023-07-14T18:10:22.457359+0000","last_unstale":"2023-07-14T18:10:24.089867+0000","last_undegraded":"2023-07-14T18:10:24.089867+0000","last_fullsized":"2023-07-14T18:10:24.089867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:55.574003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'4","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:20.500023+0000","last_change":"2023-07-14T18:10:17.214663+0000","last_active":"2023-07-14T18:11:20.500023+0000","last_peered":"2023-07-14T18:11:20.500023+0000","last_clean":"2023-07-14T18:11:20.500023+0000","last_became_active":"2023-07-14T18:10:17.214355+0000","last_became_peered":"2023-07-14T18:10:17.214355+0000","last_unstale":"2023-07-14T18:11:20.500023+0000","last_undegraded":"2023-07-14T18:11:20.500023+0000","last_fullsized":"2023-07-14T18:11:20.500023+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:56.306784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088750+0000","last_change":"2023-07-14T18:10:18.124903+0000","last_active":"2023-07-14T18:10:24.088750+0000","last_peered":"2023-07-14T18:10:24.088750+0000","last_clean":"2023-07-14T18:10:24.088750+0000","last_became_active":"2023-07-14T18:10:18.124661+0000","last_became_peered":"2023-07-14T18:10:18.124661+0000","last_unstale":"2023-07-14T18:10:24.088750+0000","last_undegraded":"2023-07-14T18:10:24.088750+0000","last_fullsized":"2023-07-14T18:10:24.088750+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:21.959726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846209+0000","last_change":"2023-07-14T18:10:22.459746+0000","last_active":"2023-07-14T18:10:23.846209+0000","last_peered":"2023-07-14T18:10:23.846209+0000","last_clean":"2023-07-14T18:10:23.846209+0000","last_became_active":"2023-07-14T18:10:22.459331+0000","last_became_peered":"2023-07-14T18:10:22.459331+0000","last_unstale":"2023-07-14T18:10:23.846209+0000","last_undegraded":"2023-07-14T18:10:23.846209+0000","last_fullsized":"2023-07-14T18:10:23.846209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:34.082947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430317+0000","last_change":"2023-07-14T18:10:17.213362+0000","last_active":"2023-07-14T18:10:37.430317+0000","last_peered":"2023-07-14T18:10:37.430317+0000","last_clean":"2023-07-14T18:10:37.430317+0000","last_became_active":"2023-07-14T18:10:17.212914+0000","last_became_peered":"2023-07-14T18:10:17.212914+0000","last_unstale":"2023-07-14T18:10:37.430317+0000","last_undegraded":"2023-07-14T18:10:37.430317+0000","last_fullsized":"2023-07-14T18:10:37.430317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:08.051946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090127+0000","last_change":"2023-07-14T18:10:18.106583+0000","last_active":"2023-07-14T18:10:24.090127+0000","last_peered":"2023-07-14T18:10:24.090127+0000","last_clean":"2023-07-14T18:10:24.090127+0000","last_became_active":"2023-07-14T18:10:18.106216+0000","last_became_peered":"2023-07-14T18:10:18.106216+0000","last_unstale":"2023-07-14T18:10:24.090127+0000","last_undegraded":"2023-07-14T18:10:24.090127+0000","last_fullsized":"2023-07-14T18:10:24.090127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:03:32.095853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089940+0000","last_change":"2023-07-14T18:10:22.447133+0000","last_active":"2023-07-14T18:10:24.089940+0000","last_peered":"2023-07-14T18:10:24.089940+0000","last_clean":"2023-07-14T18:10:24.089940+0000","last_became_active":"2023-07-14T18:10:22.446716+0000","last_became_peered":"2023-07-14T18:10:22.446716+0000","last_unstale":"2023-07-14T18:10:24.089940+0000","last_undegraded":"2023-07-14T18:10:24.089940+0000","last_fullsized":"2023-07-14T18:10:24.089940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:49.677906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.419316+0000","last_change":"2023-07-14T18:10:17.210729+0000","last_active":"2023-07-14T18:10:37.419316+0000","last_peered":"2023-07-14T18:10:37.419316+0000","last_clean":"2023-07-14T18:10:37.419316+0000","last_became_active":"2023-07-14T18:10:17.210449+0000","last_became_peered":"2023-07-14T18:10:17.210449+0000","last_unstale":"2023-07-14T18:10:37.419316+0000","last_undegraded":"2023-07-14T18:10:37.419316+0000","last_fullsized":"2023-07-14T18:10:37.419316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:57.909641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089476+0000","last_change":"2023-07-14T18:10:18.107118+0000","last_active":"2023-07-14T18:10:24.089476+0000","last_peered":"2023-07-14T18:10:24.089476+0000","last_clean":"2023-07-14T18:10:24.089476+0000","last_became_active":"2023-07-14T18:10:18.106367+0000","last_became_peered":"2023-07-14T18:10:18.106367+0000","last_unstale":"2023-07-14T18:10:24.089476+0000","last_undegraded":"2023-07-14T18:10:24.089476+0000","last_fullsized":"2023-07-14T18:10:24.089476+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:50:19.299272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438592+0000","last_change":"2023-07-14T18:10:22.500692+0000","last_active":"2023-07-14T18:10:23.438592+0000","last_peered":"2023-07-14T18:10:23.438592+0000","last_clean":"2023-07-14T18:10:23.438592+0000","last_became_active":"2023-07-14T18:10:22.500239+0000","last_became_peered":"2023-07-14T18:10:22.500239+0000","last_unstale":"2023-07-14T18:10:23.438592+0000","last_undegraded":"2023-07-14T18:10:23.438592+0000","last_fullsized":"2023-07-14T18:10:23.438592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:06.716562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089450+0000","last_change":"2023-07-14T18:10:17.212519+0000","last_active":"2023-07-14T18:10:24.089450+0000","last_peered":"2023-07-14T18:10:24.089450+0000","last_clean":"2023-07-14T18:10:24.089450+0000","last_became_active":"2023-07-14T18:10:17.212228+0000","last_became_peered":"2023-07-14T18:10:17.212228+0000","last_unstale":"2023-07-14T18:10:24.089450+0000","last_undegraded":"2023-07-14T18:10:24.089450+0000","last_fullsized":"2023-07-14T18:10:24.089450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:25:19.995310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734905+0000","last_change":"2023-07-14T18:10:18.111808+0000","last_active":"2023-07-14T18:10:23.734905+0000","last_peered":"2023-07-14T18:10:23.734905+0000","last_clean":"2023-07-14T18:10:23.734905+0000","last_became_active":"2023-07-14T18:10:18.111501+0000","last_became_peered":"2023-07-14T18:10:18.111501+0000","last_unstale":"2023-07-14T18:10:23.734905+0000","last_undegraded":"2023-07-14T18:10:23.734905+0000","last_fullsized":"2023-07-14T18:10:23.734905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:36.111363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845939+0000","last_change":"2023-07-14T18:10:22.453153+0000","last_active":"2023-07-14T18:10:23.845939+0000","last_peered":"2023-07-14T18:10:23.845939+0000","last_clean":"2023-07-14T18:10:23.845939+0000","last_became_active":"2023-07-14T18:10:22.452767+0000","last_became_peered":"2023-07-14T18:10:22.452767+0000","last_unstale":"2023-07-14T18:10:23.845939+0000","last_undegraded":"2023-07-14T18:10:23.845939+0000","last_fullsized":"2023-07-14T18:10:23.845939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:10:07.917749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089769+0000","last_change":"2023-07-14T18:10:17.215950+0000","last_active":"2023-07-14T18:10:24.089769+0000","last_peered":"2023-07-14T18:10:24.089769+0000","last_clean":"2023-07-14T18:10:24.089769+0000","last_became_active":"2023-07-14T18:10:17.215690+0000","last_became_peered":"2023-07-14T18:10:17.215690+0000","last_unstale":"2023-07-14T18:10:24.089769+0000","last_undegraded":"2023-07-14T18:10:24.089769+0000","last_fullsized":"2023-07-14T18:10:24.089769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:54.597911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734912+0000","last_change":"2023-07-14T18:10:18.114043+0000","last_active":"2023-07-14T18:10:23.734912+0000","last_peered":"2023-07-14T18:10:23.734912+0000","last_clean":"2023-07-14T18:10:23.734912+0000","last_became_active":"2023-07-14T18:10:18.113745+0000","last_became_peered":"2023-07-14T18:10:18.113745+0000","last_unstale":"2023-07-14T18:10:23.734912+0000","last_undegraded":"2023-07-14T18:10:23.734912+0000","last_fullsized":"2023-07-14T18:10:23.734912+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:42:18.691206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846054+0000","last_change":"2023-07-14T18:10:22.501904+0000","last_active":"2023-07-14T18:10:23.846054+0000","last_peered":"2023-07-14T18:10:23.846054+0000","last_clean":"2023-07-14T18:10:23.846054+0000","last_became_active":"2023-07-14T18:10:22.501500+0000","last_became_peered":"2023-07-14T18:10:22.501500+0000","last_unstale":"2023-07-14T18:10:23.846054+0000","last_undegraded":"2023-07-14T18:10:23.846054+0000","last_fullsized":"2023-07-14T18:10:23.846054+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:27:26.164775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090043+0000","last_change":"2023-07-14T18:10:18.114655+0000","last_active":"2023-07-14T18:10:24.090043+0000","last_peered":"2023-07-14T18:10:24.090043+0000","last_clean":"2023-07-14T18:10:24.090043+0000","last_became_active":"2023-07-14T18:10:18.113984+0000","last_became_peered":"2023-07-14T18:10:18.113984+0000","last_unstale":"2023-07-14T18:10:24.090043+0000","last_undegraded":"2023-07-14T18:10:24.090043+0000","last_fullsized":"2023-07-14T18:10:24.090043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:51:36.108289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:17.495424+0000","last_change":"2023-07-14T18:10:17.212612+0000","last_active":"2023-07-14T18:11:17.495424+0000","last_peered":"2023-07-14T18:11:17.495424+0000","last_clean":"2023-07-14T18:11:17.495424+0000","last_became_active":"2023-07-14T18:10:17.212282+0000","last_became_peered":"2023-07-14T18:10:17.212282+0000","last_unstale":"2023-07-14T18:11:17.495424+0000","last_undegraded":"2023-07-14T18:11:17.495424+0000","last_fullsized":"2023-07-14T18:11:17.495424+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:12:46.994247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089136+0000","last_change":"2023-07-14T18:10:22.449831+0000","last_active":"2023-07-14T18:10:24.089136+0000","last_peered":"2023-07-14T18:10:24.089136+0000","last_clean":"2023-07-14T18:10:24.089136+0000","last_became_active":"2023-07-14T18:10:22.449380+0000","last_became_peered":"2023-07-14T18:10:22.449380+0000","last_unstale":"2023-07-14T18:10:24.089136+0000","last_undegraded":"2023-07-14T18:10:24.089136+0000","last_fullsized":"2023-07-14T18:10:24.089136+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:33.180987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088819+0000","last_change":"2023-07-14T18:10:18.107041+0000","last_active":"2023-07-14T18:10:24.088819+0000","last_peered":"2023-07-14T18:10:24.088819+0000","last_clean":"2023-07-14T18:10:24.088819+0000","last_became_active":"2023-07-14T18:10:18.106715+0000","last_became_peered":"2023-07-14T18:10:18.106715+0000","last_unstale":"2023-07-14T18:10:24.088819+0000","last_undegraded":"2023-07-14T18:10:24.088819+0000","last_fullsized":"2023-07-14T18:10:24.088819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:49.905307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428953+0000","last_change":"2023-07-14T18:10:17.205525+0000","last_active":"2023-07-14T18:10:37.428953+0000","last_peered":"2023-07-14T18:10:37.428953+0000","last_clean":"2023-07-14T18:10:37.428953+0000","last_became_active":"2023-07-14T18:10:17.205164+0000","last_became_peered":"2023-07-14T18:10:17.205164+0000","last_unstale":"2023-07-14T18:10:37.428953+0000","last_undegraded":"2023-07-14T18:10:37.428953+0000","last_fullsized":"2023-07-14T18:10:37.428953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:06:06.720104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088564+0000","last_change":"2023-07-14T18:09:18.495776+0000","last_active":"2023-07-14T18:10:24.088564+0000","last_peered":"2023-07-14T18:10:24.088564+0000","last_clean":"2023-07-14T18:10:24.088564+0000","last_became_active":"2023-07-14T18:09:18.495326+0000","last_became_peered":"2023-07-14T18:09:18.495326+0000","last_unstale":"2023-07-14T18:10:24.088564+0000","last_undegraded":"2023-07-14T18:10:24.088564+0000","last_fullsized":"2023-07-14T18:10:24.088564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:06:31.483975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:06:31.483975+0000","last_clean_scrub_stamp":"2023-07-14T18:06:31.483975+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:42:18.539340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090230+0000","last_change":"2023-07-14T18:10:22.501067+0000","last_active":"2023-07-14T18:10:24.090230+0000","last_peered":"2023-07-14T18:10:24.090230+0000","last_clean":"2023-07-14T18:10:24.090230+0000","last_became_active":"2023-07-14T18:10:22.500337+0000","last_became_peered":"2023-07-14T18:10:22.500337+0000","last_unstale":"2023-07-14T18:10:24.090230+0000","last_undegraded":"2023-07-14T18:10:24.090230+0000","last_fullsized":"2023-07-14T18:10:24.090230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:43.900787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845551+0000","last_change":"2023-07-14T18:10:18.107299+0000","last_active":"2023-07-14T18:10:23.845551+0000","last_peered":"2023-07-14T18:10:23.845551+0000","last_clean":"2023-07-14T18:10:23.845551+0000","last_became_active":"2023-07-14T18:10:18.106986+0000","last_became_peered":"2023-07-14T18:10:18.106986+0000","last_unstale":"2023-07-14T18:10:23.845551+0000","last_undegraded":"2023-07-14T18:10:23.845551+0000","last_fullsized":"2023-07-14T18:10:23.845551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:47.211264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429208+0000","last_change":"2023-07-14T18:10:17.217895+0000","last_active":"2023-07-14T18:10:37.429208+0000","last_peered":"2023-07-14T18:10:37.429208+0000","last_clean":"2023-07-14T18:10:37.429208+0000","last_became_active":"2023-07-14T18:10:17.217548+0000","last_became_peered":"2023-07-14T18:10:17.217548+0000","last_unstale":"2023-07-14T18:10:37.429208+0000","last_undegraded":"2023-07-14T18:10:37.429208+0000","last_fullsized":"2023-07-14T18:10:37.429208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:29:08.392282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088848+0000","last_change":"2023-07-14T18:10:22.453140+0000","last_active":"2023-07-14T18:10:24.088848+0000","last_peered":"2023-07-14T18:10:24.088848+0000","last_clean":"2023-07-14T18:10:24.088848+0000","last_became_active":"2023-07-14T18:10:22.452731+0000","last_became_peered":"2023-07-14T18:10:22.452731+0000","last_unstale":"2023-07-14T18:10:24.088848+0000","last_undegraded":"2023-07-14T18:10:24.088848+0000","last_fullsized":"2023-07-14T18:10:24.088848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:40.973261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980328+0000","last_change":"2023-07-14T18:10:17.213431+0000","last_active":"2023-07-14T18:10:36.980328+0000","last_peered":"2023-07-14T18:10:36.980328+0000","last_clean":"2023-07-14T18:10:36.980328+0000","last_became_active":"2023-07-14T18:10:17.213125+0000","last_became_peered":"2023-07-14T18:10:17.213125+0000","last_unstale":"2023-07-14T18:10:36.980328+0000","last_undegraded":"2023-07-14T18:10:36.980328+0000","last_fullsized":"2023-07-14T18:10:36.980328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:53:36.345583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846032+0000","last_change":"2023-07-14T18:10:18.110593+0000","last_active":"2023-07-14T18:10:23.846032+0000","last_peered":"2023-07-14T18:10:23.846032+0000","last_clean":"2023-07-14T18:10:23.846032+0000","last_became_active":"2023-07-14T18:10:18.110220+0000","last_became_peered":"2023-07-14T18:10:18.110220+0000","last_unstale":"2023-07-14T18:10:23.846032+0000","last_undegraded":"2023-07-14T18:10:23.846032+0000","last_fullsized":"2023-07-14T18:10:23.846032+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:58:18.893324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088540+0000","last_change":"2023-07-14T18:10:22.500809+0000","last_active":"2023-07-14T18:10:24.088540+0000","last_peered":"2023-07-14T18:10:24.088540+0000","last_clean":"2023-07-14T18:10:24.088540+0000","last_became_active":"2023-07-14T18:10:22.500387+0000","last_became_peered":"2023-07-14T18:10:22.500387+0000","last_unstale":"2023-07-14T18:10:24.088540+0000","last_undegraded":"2023-07-14T18:10:24.088540+0000","last_fullsized":"2023-07-14T18:10:24.088540+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:26.241462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'5","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:41.563706+0000","last_change":"2023-07-14T18:10:17.209632+0000","last_active":"2023-07-14T18:10:41.563706+0000","last_peered":"2023-07-14T18:10:41.563706+0000","last_clean":"2023-07-14T18:10:41.563706+0000","last_became_active":"2023-07-14T18:10:17.209274+0000","last_became_peered":"2023-07-14T18:10:17.209274+0000","last_unstale":"2023-07-14T18:10:41.563706+0000","last_undegraded":"2023-07-14T18:10:41.563706+0000","last_fullsized":"2023-07-14T18:10:41.563706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:49:02.203177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089197+0000","last_change":"2023-07-14T18:10:18.119828+0000","last_active":"2023-07-14T18:10:24.089197+0000","last_peered":"2023-07-14T18:10:24.089197+0000","last_clean":"2023-07-14T18:10:24.089197+0000","last_became_active":"2023-07-14T18:10:18.119551+0000","last_became_peered":"2023-07-14T18:10:18.119551+0000","last_unstale":"2023-07-14T18:10:24.089197+0000","last_undegraded":"2023-07-14T18:10:24.089197+0000","last_fullsized":"2023-07-14T18:10:24.089197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:59:11.985338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549754+0000","last_change":"2023-07-14T18:10:22.450325+0000","last_active":"2023-07-14T18:10:23.549754+0000","last_peered":"2023-07-14T18:10:23.549754+0000","last_clean":"2023-07-14T18:10:23.549754+0000","last_became_active":"2023-07-14T18:10:22.449492+0000","last_became_peered":"2023-07-14T18:10:22.449492+0000","last_unstale":"2023-07-14T18:10:23.549754+0000","last_undegraded":"2023-07-14T18:10:23.549754+0000","last_fullsized":"2023-07-14T18:10:23.549754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:37:37.829472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981938+0000","last_change":"2023-07-14T18:10:17.212891+0000","last_active":"2023-07-14T18:10:36.981938+0000","last_peered":"2023-07-14T18:10:36.981938+0000","last_clean":"2023-07-14T18:10:36.981938+0000","last_became_active":"2023-07-14T18:10:17.212573+0000","last_became_peered":"2023-07-14T18:10:17.212573+0000","last_unstale":"2023-07-14T18:10:36.981938+0000","last_undegraded":"2023-07-14T18:10:36.981938+0000","last_fullsized":"2023-07-14T18:10:36.981938+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:38:07.247348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440729+0000","last_change":"2023-07-14T18:10:18.113389+0000","last_active":"2023-07-14T18:10:23.440729+0000","last_peered":"2023-07-14T18:10:23.440729+0000","last_clean":"2023-07-14T18:10:23.440729+0000","last_became_active":"2023-07-14T18:10:18.113053+0000","last_became_peered":"2023-07-14T18:10:18.113053+0000","last_unstale":"2023-07-14T18:10:23.440729+0000","last_undegraded":"2023-07-14T18:10:23.440729+0000","last_fullsized":"2023-07-14T18:10:23.440729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:29:31.077735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438597+0000","last_change":"2023-07-14T18:10:22.500749+0000","last_active":"2023-07-14T18:10:23.438597+0000","last_peered":"2023-07-14T18:10:23.438597+0000","last_clean":"2023-07-14T18:10:23.438597+0000","last_became_active":"2023-07-14T18:10:22.500357+0000","last_became_peered":"2023-07-14T18:10:22.500357+0000","last_unstale":"2023-07-14T18:10:23.438597+0000","last_undegraded":"2023-07-14T18:10:23.438597+0000","last_fullsized":"2023-07-14T18:10:23.438597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:05:10.947498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980089+0000","last_change":"2023-07-14T18:10:17.212501+0000","last_active":"2023-07-14T18:10:36.980089+0000","last_peered":"2023-07-14T18:10:36.980089+0000","last_clean":"2023-07-14T18:10:36.980089+0000","last_became_active":"2023-07-14T18:10:17.212171+0000","last_became_peered":"2023-07-14T18:10:17.212171+0000","last_unstale":"2023-07-14T18:10:36.980089+0000","last_undegraded":"2023-07-14T18:10:36.980089+0000","last_fullsized":"2023-07-14T18:10:36.980089+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:37.208644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438420+0000","last_change":"2023-07-14T18:10:18.111692+0000","last_active":"2023-07-14T18:10:23.438420+0000","last_peered":"2023-07-14T18:10:23.438420+0000","last_clean":"2023-07-14T18:10:23.438420+0000","last_became_active":"2023-07-14T18:10:18.111424+0000","last_became_peered":"2023-07-14T18:10:18.111424+0000","last_unstale":"2023-07-14T18:10:23.438420+0000","last_undegraded":"2023-07-14T18:10:23.438420+0000","last_fullsized":"2023-07-14T18:10:23.438420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:06.610911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089989+0000","last_change":"2023-07-14T18:10:22.456773+0000","last_active":"2023-07-14T18:10:24.089989+0000","last_peered":"2023-07-14T18:10:24.089989+0000","last_clean":"2023-07-14T18:10:24.089989+0000","last_became_active":"2023-07-14T18:10:22.456309+0000","last_became_peered":"2023-07-14T18:10:22.456309+0000","last_unstale":"2023-07-14T18:10:24.089989+0000","last_undegraded":"2023-07-14T18:10:24.089989+0000","last_fullsized":"2023-07-14T18:10:24.089989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:29:09.644386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980843+0000","last_change":"2023-07-14T18:10:17.205109+0000","last_active":"2023-07-14T18:10:36.980843+0000","last_peered":"2023-07-14T18:10:36.980843+0000","last_clean":"2023-07-14T18:10:36.980843+0000","last_became_active":"2023-07-14T18:10:17.204644+0000","last_became_peered":"2023-07-14T18:10:17.204644+0000","last_unstale":"2023-07-14T18:10:36.980843+0000","last_undegraded":"2023-07-14T18:10:36.980843+0000","last_fullsized":"2023-07-14T18:10:36.980843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:08.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088692+0000","last_change":"2023-07-14T18:10:18.109143+0000","last_active":"2023-07-14T18:10:24.088692+0000","last_peered":"2023-07-14T18:10:24.088692+0000","last_clean":"2023-07-14T18:10:24.088692+0000","last_became_active":"2023-07-14T18:10:18.108841+0000","last_became_peered":"2023-07-14T18:10:18.108841+0000","last_unstale":"2023-07-14T18:10:24.088692+0000","last_undegraded":"2023-07-14T18:10:24.088692+0000","last_fullsized":"2023-07-14T18:10:24.088692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:30:16.212954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438346+0000","last_change":"2023-07-14T18:10:22.450564+0000","last_active":"2023-07-14T18:10:23.438346+0000","last_peered":"2023-07-14T18:10:23.438346+0000","last_clean":"2023-07-14T18:10:23.438346+0000","last_became_active":"2023-07-14T18:10:22.449801+0000","last_became_peered":"2023-07-14T18:10:22.449801+0000","last_unstale":"2023-07-14T18:10:23.438346+0000","last_undegraded":"2023-07-14T18:10:23.438346+0000","last_fullsized":"2023-07-14T18:10:23.438346+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:00:04.711449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549788+0000","last_change":"2023-07-14T18:10:17.211522+0000","last_active":"2023-07-14T18:10:23.549788+0000","last_peered":"2023-07-14T18:10:23.549788+0000","last_clean":"2023-07-14T18:10:23.549788+0000","last_became_active":"2023-07-14T18:10:17.211163+0000","last_became_peered":"2023-07-14T18:10:17.211163+0000","last_unstale":"2023-07-14T18:10:23.549788+0000","last_undegraded":"2023-07-14T18:10:23.549788+0000","last_fullsized":"2023-07-14T18:10:23.549788+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:14:47.230105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089155+0000","last_change":"2023-07-14T18:10:18.111885+0000","last_active":"2023-07-14T18:10:24.089155+0000","last_peered":"2023-07-14T18:10:24.089155+0000","last_clean":"2023-07-14T18:10:24.089155+0000","last_became_active":"2023-07-14T18:10:18.111520+0000","last_became_peered":"2023-07-14T18:10:18.111520+0000","last_unstale":"2023-07-14T18:10:24.089155+0000","last_undegraded":"2023-07-14T18:10:24.089155+0000","last_fullsized":"2023-07-14T18:10:24.089155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:06:29.717829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438617+0000","last_change":"2023-07-14T18:10:22.500612+0000","last_active":"2023-07-14T18:10:23.438617+0000","last_peered":"2023-07-14T18:10:23.438617+0000","last_clean":"2023-07-14T18:10:23.438617+0000","last_became_active":"2023-07-14T18:10:22.500219+0000","last_became_peered":"2023-07-14T18:10:22.500219+0000","last_unstale":"2023-07-14T18:10:23.438617+0000","last_undegraded":"2023-07-14T18:10:23.438617+0000","last_fullsized":"2023-07-14T18:10:23.438617+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:09.251191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981056+0000","last_change":"2023-07-14T18:10:17.214307+0000","last_active":"2023-07-14T18:10:36.981056+0000","last_peered":"2023-07-14T18:10:36.981056+0000","last_clean":"2023-07-14T18:10:36.981056+0000","last_became_active":"2023-07-14T18:10:17.214004+0000","last_became_peered":"2023-07-14T18:10:17.214004+0000","last_unstale":"2023-07-14T18:10:36.981056+0000","last_undegraded":"2023-07-14T18:10:36.981056+0000","last_fullsized":"2023-07-14T18:10:36.981056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:50:59.158356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089823+0000","last_change":"2023-07-14T18:10:18.120504+0000","last_active":"2023-07-14T18:10:24.089823+0000","last_peered":"2023-07-14T18:10:24.089823+0000","last_clean":"2023-07-14T18:10:24.089823+0000","last_became_active":"2023-07-14T18:10:18.120133+0000","last_became_peered":"2023-07-14T18:10:18.120133+0000","last_unstale":"2023-07-14T18:10:24.089823+0000","last_undegraded":"2023-07-14T18:10:24.089823+0000","last_fullsized":"2023-07-14T18:10:24.089823+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:54.324914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089420+0000","last_change":"2023-07-14T18:10:22.453040+0000","last_active":"2023-07-14T18:10:24.089420+0000","last_peered":"2023-07-14T18:10:24.089420+0000","last_clean":"2023-07-14T18:10:24.089420+0000","last_became_active":"2023-07-14T18:10:22.452694+0000","last_became_peered":"2023-07-14T18:10:22.452694+0000","last_unstale":"2023-07-14T18:10:24.089420+0000","last_undegraded":"2023-07-14T18:10:24.089420+0000","last_fullsized":"2023-07-14T18:10:24.089420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:43:06.299843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428686+0000","last_change":"2023-07-14T18:10:17.212480+0000","last_active":"2023-07-14T18:10:37.428686+0000","last_peered":"2023-07-14T18:10:37.428686+0000","last_clean":"2023-07-14T18:10:37.428686+0000","last_became_active":"2023-07-14T18:10:17.212080+0000","last_became_peered":"2023-07-14T18:10:17.212080+0000","last_unstale":"2023-07-14T18:10:37.428686+0000","last_undegraded":"2023-07-14T18:10:37.428686+0000","last_fullsized":"2023-07-14T18:10:37.428686+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:19:44.529873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089506+0000","last_change":"2023-07-14T18:10:18.121738+0000","last_active":"2023-07-14T18:10:24.089506+0000","last_peered":"2023-07-14T18:10:24.089506+0000","last_clean":"2023-07-14T18:10:24.089506+0000","last_became_active":"2023-07-14T18:10:18.121186+0000","last_became_peered":"2023-07-14T18:10:18.121186+0000","last_unstale":"2023-07-14T18:10:24.089506+0000","last_undegraded":"2023-07-14T18:10:24.089506+0000","last_fullsized":"2023-07-14T18:10:24.089506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:40:16.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440225+0000","last_change":"2023-07-14T18:10:22.461051+0000","last_active":"2023-07-14T18:10:23.440225+0000","last_peered":"2023-07-14T18:10:23.440225+0000","last_clean":"2023-07-14T18:10:23.440225+0000","last_became_active":"2023-07-14T18:10:22.460663+0000","last_became_peered":"2023-07-14T18:10:22.460663+0000","last_unstale":"2023-07-14T18:10:23.440225+0000","last_undegraded":"2023-07-14T18:10:23.440225+0000","last_fullsized":"2023-07-14T18:10:23.440225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:58:08.355441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090658+0000","last_change":"2023-07-14T18:10:17.211686+0000","last_active":"2023-07-14T18:10:24.090658+0000","last_peered":"2023-07-14T18:10:24.090658+0000","last_clean":"2023-07-14T18:10:24.090658+0000","last_became_active":"2023-07-14T18:10:17.211367+0000","last_became_peered":"2023-07-14T18:10:17.211367+0000","last_unstale":"2023-07-14T18:10:24.090658+0000","last_undegraded":"2023-07-14T18:10:24.090658+0000","last_fullsized":"2023-07-14T18:10:24.090658+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:50.829623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089608+0000","last_change":"2023-07-14T18:10:18.116960+0000","last_active":"2023-07-14T18:10:24.089608+0000","last_peered":"2023-07-14T18:10:24.089608+0000","last_clean":"2023-07-14T18:10:24.089608+0000","last_became_active":"2023-07-14T18:10:18.116416+0000","last_became_peered":"2023-07-14T18:10:18.116416+0000","last_unstale":"2023-07-14T18:10:24.089608+0000","last_undegraded":"2023-07-14T18:10:24.089608+0000","last_fullsized":"2023-07-14T18:10:24.089608+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:33.066056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089552+0000","last_change":"2023-07-14T18:10:22.450045+0000","last_active":"2023-07-14T18:10:24.089552+0000","last_peered":"2023-07-14T18:10:24.089552+0000","last_clean":"2023-07-14T18:10:24.089552+0000","last_became_active":"2023-07-14T18:10:22.449621+0000","last_became_peered":"2023-07-14T18:10:22.449621+0000","last_unstale":"2023-07-14T18:10:24.089552+0000","last_undegraded":"2023-07-14T18:10:24.089552+0000","last_fullsized":"2023-07-14T18:10:24.089552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:48:23.837464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846188+0000","last_change":"2023-07-14T18:10:17.210691+0000","last_active":"2023-07-14T18:10:23.846188+0000","last_peered":"2023-07-14T18:10:23.846188+0000","last_clean":"2023-07-14T18:10:23.846188+0000","last_became_active":"2023-07-14T18:10:17.210391+0000","last_became_peered":"2023-07-14T18:10:17.210391+0000","last_unstale":"2023-07-14T18:10:23.846188+0000","last_undegraded":"2023-07-14T18:10:23.846188+0000","last_fullsized":"2023-07-14T18:10:23.846188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:20:04.361243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089271+0000","last_change":"2023-07-14T18:10:18.115347+0000","last_active":"2023-07-14T18:10:24.089271+0000","last_peered":"2023-07-14T18:10:24.089271+0000","last_clean":"2023-07-14T18:10:24.089271+0000","last_became_active":"2023-07-14T18:10:18.115027+0000","last_became_peered":"2023-07-14T18:10:18.115027+0000","last_unstale":"2023-07-14T18:10:24.089271+0000","last_undegraded":"2023-07-14T18:10:24.089271+0000","last_fullsized":"2023-07-14T18:10:24.089271+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:33:02.993257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088765+0000","last_change":"2023-07-14T18:10:18.103185+0000","last_active":"2023-07-14T18:10:24.088765+0000","last_peered":"2023-07-14T18:10:24.088765+0000","last_clean":"2023-07-14T18:10:24.088765+0000","last_became_active":"2023-07-14T18:10:18.102800+0000","last_became_peered":"2023-07-14T18:10:18.102800+0000","last_unstale":"2023-07-14T18:10:24.088765+0000","last_undegraded":"2023-07-14T18:10:24.088765+0000","last_fullsized":"2023-07-14T18:10:24.088765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:35:22.321627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550057+0000","last_change":"2023-07-14T18:10:18.113501+0000","last_active":"2023-07-14T18:10:23.550057+0000","last_peered":"2023-07-14T18:10:23.550057+0000","last_clean":"2023-07-14T18:10:23.550057+0000","last_became_active":"2023-07-14T18:10:18.113205+0000","last_became_peered":"2023-07-14T18:10:18.113205+0000","last_unstale":"2023-07-14T18:10:23.550057+0000","last_undegraded":"2023-07-14T18:10:23.550057+0000","last_fullsized":"2023-07-14T18:10:23.550057+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:13:56.420803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549722+0000","last_change":"2023-07-14T18:10:18.114282+0000","last_active":"2023-07-14T18:10:23.549722+0000","last_peered":"2023-07-14T18:10:23.549722+0000","last_clean":"2023-07-14T18:10:23.549722+0000","last_became_active":"2023-07-14T18:10:18.113968+0000","last_became_peered":"2023-07-14T18:10:18.113968+0000","last_unstale":"2023-07-14T18:10:23.549722+0000","last_undegraded":"2023-07-14T18:10:23.549722+0000","last_fullsized":"2023-07-14T18:10:23.549722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:36.356073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440436+0000","last_change":"2023-07-14T18:10:18.116781+0000","last_active":"2023-07-14T18:10:23.440436+0000","last_peered":"2023-07-14T18:10:23.440436+0000","last_clean":"2023-07-14T18:10:23.440436+0000","last_became_active":"2023-07-14T18:10:18.116352+0000","last_became_peered":"2023-07-14T18:10:18.116352+0000","last_unstale":"2023-07-14T18:10:23.440436+0000","last_undegraded":"2023-07-14T18:10:23.440436+0000","last_fullsized":"2023-07-14T18:10:23.440436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:48:03.670475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846088+0000","last_change":"2023-07-14T18:10:18.112291+0000","last_active":"2023-07-14T18:10:23.846088+0000","last_peered":"2023-07-14T18:10:23.846088+0000","last_clean":"2023-07-14T18:10:23.846088+0000","last_became_active":"2023-07-14T18:10:18.111989+0000","last_became_peered":"2023-07-14T18:10:18.111989+0000","last_unstale":"2023-07-14T18:10:23.846088+0000","last_undegraded":"2023-07-14T18:10:23.846088+0000","last_fullsized":"2023-07-14T18:10:23.846088+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:11:33.924006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089268+0000","last_change":"2023-07-14T18:10:18.112316+0000","last_active":"2023-07-14T18:10:24.089268+0000","last_peered":"2023-07-14T18:10:24.089268+0000","last_clean":"2023-07-14T18:10:24.089268+0000","last_became_active":"2023-07-14T18:10:18.112030+0000","last_became_peered":"2023-07-14T18:10:18.112030+0000","last_unstale":"2023-07-14T18:10:24.089268+0000","last_undegraded":"2023-07-14T18:10:24.089268+0000","last_fullsized":"2023-07-14T18:10:24.089268+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:14:55.405097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550228+0000","last_change":"2023-07-14T18:10:18.116343+0000","last_active":"2023-07-14T18:10:23.550228+0000","last_peered":"2023-07-14T18:10:23.550228+0000","last_clean":"2023-07-14T18:10:23.550228+0000","last_became_active":"2023-07-14T18:10:18.116041+0000","last_became_peered":"2023-07-14T18:10:18.116041+0000","last_unstale":"2023-07-14T18:10:23.550228+0000","last_undegraded":"2023-07-14T18:10:23.550228+0000","last_fullsized":"2023-07-14T18:10:23.550228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:43:43.627634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089357+0000","last_change":"2023-07-14T18:10:18.117580+0000","last_active":"2023-07-14T18:10:24.089357+0000","last_peered":"2023-07-14T18:10:24.089357+0000","last_clean":"2023-07-14T18:10:24.089357+0000","last_became_active":"2023-07-14T18:10:18.117249+0000","last_became_peered":"2023-07-14T18:10:18.117249+0000","last_unstale":"2023-07-14T18:10:24.089357+0000","last_undegraded":"2023-07-14T18:10:24.089357+0000","last_fullsized":"2023-07-14T18:10:24.089357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:42.562595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089822+0000","last_change":"2023-07-14T18:10:18.107276+0000","last_active":"2023-07-14T18:10:24.089822+0000","last_peered":"2023-07-14T18:10:24.089822+0000","last_clean":"2023-07-14T18:10:24.089822+0000","last_became_active":"2023-07-14T18:10:18.107005+0000","last_became_peered":"2023-07-14T18:10:18.107005+0000","last_unstale":"2023-07-14T18:10:24.089822+0000","last_undegraded":"2023-07-14T18:10:24.089822+0000","last_fullsized":"2023-07-14T18:10:24.089822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:13.352444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089145+0000","last_change":"2023-07-14T18:10:18.107921+0000","last_active":"2023-07-14T18:10:24.089145+0000","last_peered":"2023-07-14T18:10:24.089145+0000","last_clean":"2023-07-14T18:10:24.089145+0000","last_became_active":"2023-07-14T18:10:18.107637+0000","last_became_peered":"2023-07-14T18:10:18.107637+0000","last_unstale":"2023-07-14T18:10:24.089145+0000","last_undegraded":"2023-07-14T18:10:24.089145+0000","last_fullsized":"2023-07-14T18:10:24.089145+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:32.687817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089709+0000","last_change":"2023-07-14T18:10:18.120125+0000","last_active":"2023-07-14T18:10:24.089709+0000","last_peered":"2023-07-14T18:10:24.089709+0000","last_clean":"2023-07-14T18:10:24.089709+0000","last_became_active":"2023-07-14T18:10:18.119814+0000","last_became_peered":"2023-07-14T18:10:18.119814+0000","last_unstale":"2023-07-14T18:10:24.089709+0000","last_undegraded":"2023-07-14T18:10:24.089709+0000","last_fullsized":"2023-07-14T18:10:24.089709+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:37:22.531320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845885+0000","last_change":"2023-07-14T18:10:18.113155+0000","last_active":"2023-07-14T18:10:23.845885+0000","last_peered":"2023-07-14T18:10:23.845885+0000","last_clean":"2023-07-14T18:10:23.845885+0000","last_became_active":"2023-07-14T18:10:18.112837+0000","last_became_peered":"2023-07-14T18:10:18.112837+0000","last_unstale":"2023-07-14T18:10:23.845885+0000","last_undegraded":"2023-07-14T18:10:23.845885+0000","last_fullsized":"2023-07-14T18:10:23.845885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:56:13.124803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088936+0000","last_change":"2023-07-14T18:10:18.112290+0000","last_active":"2023-07-14T18:10:24.088936+0000","last_peered":"2023-07-14T18:10:24.088936+0000","last_clean":"2023-07-14T18:10:24.088936+0000","last_became_active":"2023-07-14T18:10:18.111988+0000","last_became_peered":"2023-07-14T18:10:18.111988+0000","last_unstale":"2023-07-14T18:10:24.088936+0000","last_undegraded":"2023-07-14T18:10:24.088936+0000","last_fullsized":"2023-07-14T18:10:24.088936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:26.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62845,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":290,"num_read_kb":209,"num_write":154,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":188535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449112,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":470478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.36399999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.314},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":4,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.66400000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.435}]},{"osd":6,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.55500000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.52500000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.496}]},{"osd":10,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.76100000000000001}]}]},{"osd":10,"up_from":71,"seq":304942678044,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1116308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.57399999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77000000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82099999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64500000000000002,"15min":0.64400000000000002},"min":{"1min":0.435,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86199999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52900000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.85099999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.42299999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.35299999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89500000000000002,"5min":1.042,"15min":1.042},"last":0.35299999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71599999999999997,"15min":0.72099999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96199999999999997,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.73199999999999998,"15min":0.73499999999999999},"min":{"1min":0.44900000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.44900000000000001}]},{"osd":6,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.73699999999999999,"15min":0.748},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.94399999999999995,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.76700000000000002,"15min":0.77300000000000002},"min":{"1min":0.55200000000000005,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.036,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.69299999999999995}]},{"osd":8,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.79800000000000004,"15min":0.80200000000000005},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.81100000000000005,"15min":0.82099999999999995},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.0229999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.53800000000000003}]},{"osd":9,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.80700000000000005,"15min":0.81100000000000005},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1399999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.81599999999999995,"15min":0.82999999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.069,"5min":1.329,"15min":1.329},"last":0.623}]},{"osd":11,"last update":"Fri Jul 14 18:10:41 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.66200000000000003}]}]},{"osd":9,"up_from":66,"seq":283467841569,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":474810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.442,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.53600000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.44700000000000001,"15min":0.45800000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.28699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.38200000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83699999999999997,"5min":0.875,"15min":0.875},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82899999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.42499999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.59599999999999997,"15min":0.60299999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81999999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.57199999999999995}]},{"osd":3,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.496,"15min":0.503},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68400000000000005,"5min":0.75,"15min":0.75},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.51000000000000001,"15min":0.52200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66800000000000004,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.313}]},{"osd":4,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.63500000000000001,"15min":0.64700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76800000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.373},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67500000000000004,"15min":0.68899999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.434}]},{"osd":5,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64900000000000002,"15min":0.65800000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.91000000000000003,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.68200000000000005,"15min":0.69799999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.80200000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58199999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.71899999999999997,"15min":0.73499999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.72299999999999998,"15min":0.73899999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.871,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.44800000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.72999999999999998,"15min":0.748},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.83199999999999996,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73799999999999999,"15min":0.749},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.97999999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.64900000000000002}]},{"osd":10,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.74399999999999999,"15min":0.76100000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.871,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.73699999999999999,"15min":0.75600000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.52300000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":8,"up_from":58,"seq":249108103204,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":470303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.41499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47899999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.82999999999999996}]},{"osd":2,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.42999999999999999,"15min":0.42299999999999999},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46899999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.57099999999999995}]},{"osd":3,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.433,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79700000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.50800000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64100000000000001,"15min":0.63900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.36299999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.64800000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.52300000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72099999999999997,"15min":0.72599999999999998},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.53100000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.746,"5min":0.70099999999999996,"15min":0.69299999999999995},"min":{"1min":0.501,"5min":0.439,"15min":0.439},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.79900000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.70299999999999996,"15min":0.69099999999999995},"min":{"1min":0.47299999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.046,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.73599999999999999,"15min":0.72799999999999998},"min":{"1min":0.55500000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0649999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.77100000000000002}]},{"osd":10,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.91600000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]}]},{"osd":7,"up_from":50,"seq":214748364841,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1066836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.41399999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.42799999999999999,"15min":0.41899999999999998},"min":{"1min":0.314,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.373},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.45100000000000001,"15min":0.44},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.59799999999999998,"15min":0.58099999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60299999999999998,"15min":0.58699999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.441}]},{"osd":3,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.55900000000000005,"15min":0.54300000000000004},"min":{"1min":0.41099999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75800000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.59099999999999997,"15min":0.58099999999999996},"min":{"1min":0.43099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.77400000000000002,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34300000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77800000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.5}]},{"osd":5,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.65600000000000003,"15min":0.63200000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.68600000000000005,"15min":0.68000000000000005},"min":{"1min":0.47099999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.42799999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.86299999999999999,"5min":1.024,"15min":1.024},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.67300000000000004,"15min":0.65100000000000002},"min":{"1min":0.52000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91000000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52000000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:10:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.66700000000000004,"15min":0.67300000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.877,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.39900000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72799999999999998,"15min":0.73199999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.51100000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72799999999999998,"15min":0.72499999999999998},"min":{"1min":0.49199999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96599999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77400000000000002,"15min":0.76100000000000001},"min":{"1min":0.52000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.58199999999999996}]},{"osd":11,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.56499999999999995}]}]},{"osd":6,"up_from":45,"seq":193273528364,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.5},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.27700000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.627,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.442,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.64200000000000002}]},{"osd":2,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58499999999999996,"15min":0.58099999999999996},"min":{"1min":0.438,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82299999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.622,"15min":0.62},"min":{"1min":0.45900000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78800000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.66600000000000004}]},{"osd":3,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.627},{"interface":"front","average":{"1min":0.439,"5min":0.45300000000000001,"15min":0.46000000000000002},"min":{"1min":0.30099999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83699999999999997,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.61699999999999999,"15min":0.60999999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83899999999999997,"5min":0.879,"15min":0.879},"last":0.78300000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.625,"15min":0.629},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64000000000000001,"15min":0.63700000000000001},"min":{"1min":0.46999999999999997,"5min":0.433,"15min":0.433},"max":{"1min":0.88,"5min":1.012,"15min":1.012},"last":0.74299999999999999}]},{"osd":7,"last update":"Fri Jul 14 18:10:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.44600000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.94299999999999995,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.877},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70299999999999996,"15min":0.70199999999999996},"min":{"1min":0.51100000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.85999999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:10:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.75,"15min":0.76200000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92200000000000004,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.68200000000000005,"15min":0.69199999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88500000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.71299999999999997}]},{"osd":9,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.39300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.85399999999999998,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61299999999999999,"15min":0.61899999999999999},"min":{"1min":0.40400000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82199999999999995,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59599999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:10:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82599999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":524351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.48499999999999999},"min":{"1min":0.36499999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80400000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.503},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.495},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.62}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.622,"15min":0.60399999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.97799999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.45300000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80600000000000005,"5min":1.337,"15min":1.337},"last":0.82799999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:11:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56499999999999995,"15min":0.52300000000000002},"min":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.90700000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54100000000000004,"15min":0.50700000000000001},"min":{"1min":0.41499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.54700000000000004}]},{"osd":4,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.46800000000000003,"15min":0.42999999999999999},"min":{"1min":0.41099999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.48499999999999999,"15min":0.44400000000000001},"min":{"1min":0.42199999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.72599999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.47199999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92900000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.875},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.64300000000000002,"15min":0.62},"min":{"1min":0.45800000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.86599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.91900000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:11:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.627,"15min":0.58899999999999997},"min":{"1min":0.49399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96099999999999997,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.64600000000000002,"15min":0.624},"min":{"1min":0.43099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.76300000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.629,"15min":0.59199999999999997},"min":{"1min":0.51600000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.624,"15min":0.58599999999999997},"min":{"1min":0.47299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.90700000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70899999999999996,"15min":0.69599999999999995},"min":{"1min":0.59399999999999997,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66100000000000003,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.79100000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80000000000000004,"15min":0.79500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0189999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.79200000000000004,"15min":0.78400000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.93100000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.88,"5min":0.82399999999999995,"15min":0.81499999999999995},"min":{"1min":0.60699999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":1.036}]},{"osd":11,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":1.0089999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":467278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.55100000000000005,"15min":0.503},"min":{"1min":0.34599999999999997,"5min":0.312,"15min":0.312},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.55200000000000005,"15min":0.51200000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.952,"5min":2.952,"15min":2.952},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52600000000000002,"15min":0.51300000000000001},"min":{"1min":0.41299999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68500000000000005,"5min":0.997,"15min":0.997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53900000000000003,"15min":0.50800000000000001},"min":{"1min":0.40899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78900000000000003,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.58899999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.32400000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.57399999999999995,"15min":0.53200000000000003},"min":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.53100000000000003},"min":{"1min":0.44,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.495}]},{"osd":5,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70699999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.59099999999999997},"min":{"1min":0.45700000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:10:54 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49299999999999999,"15min":0.47899999999999998},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.59199999999999997,"15min":0.55500000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.6689999999999996,"5min":4.6689999999999996,"15min":4.6689999999999996},"last":0.379}]},{"osd":7,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.65700000000000003,"15min":0.65900000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.92100000000000004,"5min":0.998,"15min":0.998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.621,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85299999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.50800000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64500000000000002,"15min":0.65100000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81299999999999994,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.72099999999999997,"15min":0.72799999999999998},"min":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.88800000000000001,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.68999999999999995}]},{"osd":9,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.66500000000000004,"15min":0.68200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.66200000000000003,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.47599999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.76800000000000002,"15min":0.747},"min":{"1min":0.54500000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":4.4740000000000002,"5min":4.4740000000000002,"15min":4.4740000000000002},"last":0.621},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.60799999999999998}]},{"osd":11,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.67400000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":526043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.67500000000000004,"5min":0.995,"15min":0.995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68899999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.39000000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":1.29,"15min":1.29},"last":0.626},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.34899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.01,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.65800000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80900000000000005,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59099999999999997,"15min":0.55400000000000005},"min":{"1min":0.48299999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91700000000000004,"5min":0.996,"15min":0.996},"last":0.379}]},{"osd":4,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.57899999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.498,"15min":0.47499999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.49199999999999999,"15min":0.46400000000000002},"min":{"1min":0.29699999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.82699999999999996,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Jul 14 18:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.52000000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90000000000000002,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.66200000000000003,"15min":0.64200000000000002},"min":{"1min":0.54000000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.68600000000000005}]},{"osd":7,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73299999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67400000000000004,"15min":0.66700000000000004},"min":{"1min":0.40699999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":0.89100000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.79500000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.73599999999999999,"15min":0.72599999999999998},"min":{"1min":0.61599999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.76400000000000001,"15min":0.76200000000000001},"min":{"1min":0.52000000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.002,"5min":1.524,"15min":1.524},"last":0.69999999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.748},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.76100000000000001}]},{"osd":11,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.82599999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":465753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.36099999999999999,"15min":0.33100000000000002},"min":{"1min":0.25900000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":0.314},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.38400000000000001,"15min":0.35199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.33800000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55300000000000005,"15min":0.53500000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77300000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.36399999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.53700000000000003},"min":{"1min":0.45800000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69099999999999995,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55400000000000005,"15min":0.53500000000000003},"min":{"1min":0.42499999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.57599999999999996}]},{"osd":4,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56799999999999995,"15min":0.52800000000000002},"min":{"1min":0.38700000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.2070000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.54400000000000004},"min":{"1min":0.40699999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77800000000000002,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.52800000000000002}]},{"osd":5,"last update":"Fri Jul 14 18:10:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.626,"15min":0.61199999999999999},"min":{"1min":0.47199999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78300000000000003,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.502},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.38300000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.69199999999999995,"15min":0.65300000000000002},"min":{"1min":0.41199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.0299999999999994,"5min":9.0299999999999994,"15min":9.0299999999999994},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68899999999999995,"15min":0.68500000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.232,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.63}]},{"osd":8,"last update":"Fri Jul 14 18:09:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.64100000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.76300000000000001,"15min":0.78700000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.48699999999999999}]},{"osd":10,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.72699999999999998,"15min":0.67500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":9.0920000000000005,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.77700000000000002,"15min":0.79300000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.258,"5min":1.504,"15min":1.504},"last":0.65900000000000003}]},{"osd":11,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":8.9730000000000008,"5min":8.9730000000000008,"15min":8.9730000000000008},"last":0.60399999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953526,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":470882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78500000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.36399999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.498,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40400000000000003,"15min":0.38400000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56399999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.377}]},{"osd":2,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59599999999999997,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85399999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.56999999999999995},"min":{"1min":0.42399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56299999999999994,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71799999999999997,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.505},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40000000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81899999999999995,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.53300000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.439,"5min":0.373,"15min":0.373},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60199999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:11:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.46100000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88100000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97799999999999998,"5min":0.999,"15min":0.999},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75800000000000001,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59299999999999997,"15min":0.58199999999999996},"min":{"1min":0.47199999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.56100000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67600000000000005,"15min":0.67300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.87,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.71899999999999997,"15min":0.71299999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.70499999999999996,"15min":0.70699999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.71199999999999997,"15min":0.71599999999999997},"min":{"1min":0.57699999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.84399999999999997,"5min":1.024,"15min":1.024},"last":0.624}]},{"osd":10,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.71499999999999997,"15min":0.70899999999999996},"min":{"1min":0.52000000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.71399999999999997,"15min":0.71499999999999997},"min":{"1min":0.56599999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.82199999999999995,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.59199999999999997}]},{"osd":11,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.70399999999999996}]}]},{"osd":5,"up_from":37,"seq":158913790002,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1585152,"data_stored":1062082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86299999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.51500000000000001,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.61099999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.42699999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57799999999999996,"15min":0.56999999999999995},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.629},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56799999999999995,"15min":0.56100000000000005},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78200000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.42699999999999999,"15min":0.42299999999999999},"min":{"1min":0.30499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.29099999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72399999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59299999999999997,"15min":0.60099999999999998},"min":{"1min":0.42499999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68700000000000006,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83099999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.60399999999999998},"min":{"1min":0.432,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.90100000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75700000000000001,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.44800000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85799999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.66600000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.628,"15min":0.61799999999999999},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93300000000000005,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.66200000000000003,"15min":0.65800000000000003},"min":{"1min":0.47699999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.65500000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83899999999999997,"5min":0.873,"15min":0.873},"last":0.64200000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.66400000000000003,"15min":0.66000000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81999999999999995,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.70199999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:11:23.479 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T18:11:24.042 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:24.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:24 smithi115 ceph-mon[127451]: pgmap v284: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 7 op/s 2023-07-14T18:11:24.461 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:24 smithi115 ceph-mon[127451]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:11:24.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:24 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:24.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:24 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:24.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:24 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:11:24.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:24 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:11:24.462 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:24 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:24.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:24 smithi150 ceph-mon[128853]: pgmap v284: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 7 op/s 2023-07-14T18:11:24.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:24 smithi150 ceph-mon[128853]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:11:24.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:24 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:24.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:24 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:24.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:24 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:11:24.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:24 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:11:24.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:24 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:24.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:24 smithi187 ceph-mon[129317]: pgmap v284: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 7 op/s 2023-07-14T18:11:24.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:24 smithi187 ceph-mon[129317]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:11:24.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:24 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:24.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:24 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:24.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:24 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:11:24.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:24 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:11:24.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:24 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' 2023-07-14T18:11:25.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.249+0000 7efce4315700 1 -- 172.21.15.115:0/505374814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 msgr2=0x7efcdc1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.249+0000 7efce4315700 1 --2- 172.21.15.115:0/505374814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 0x7efcdc1037b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efccc0099c0 tx=0x7efccc009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.249+0000 7efce4315700 1 -- 172.21.15.115:0/505374814 shutdown_connections 2023-07-14T18:11:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.249+0000 7efce4315700 1 --2- 172.21.15.115:0/505374814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc103ee0 0x7efcdc10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.249+0000 7efce4315700 1 --2- 172.21.15.115:0/505374814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc0fee40 0x7efcdc0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.249+0000 7efce4315700 1 --2- 172.21.15.115:0/505374814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 0x7efcdc1037b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.249+0000 7efce4315700 1 -- 172.21.15.115:0/505374814 >> 172.21.15.115:0/505374814 conn(0x7efcdc075ca0 msgr2=0x7efcdc0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.249+0000 7efce4315700 1 -- 172.21.15.115:0/505374814 shutdown_connections 2023-07-14T18:11:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.249+0000 7efce4315700 1 -- 172.21.15.115:0/505374814 wait complete. 2023-07-14T18:11:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.250+0000 7efce4315700 1 Processor -- start 2023-07-14T18:11:25.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.250+0000 7efce4315700 1 -- start start 2023-07-14T18:11:25.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce4315700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc0fee40 0x7efcdc19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:25.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce4315700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 0x7efcdc19aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:25.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce4315700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc103ee0 0x7efcdc1949f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:25.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce4315700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc195060 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce4315700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc1951d0 con 0x7efcdc0fee40 2023-07-14T18:11:25.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce4315700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc195390 con 0x7efcdc103ee0 2023-07-14T18:11:25.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce18b0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 0x7efcdc19aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:25.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce18b0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 0x7efcdc19aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53546/0 (socket says 172.21.15.115:53546) 2023-07-14T18:11:25.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce18b0700 1 -- 172.21.15.115:0/2875170951 learned_addr learned my addr 172.21.15.115:0/2875170951 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:25.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce28b2700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc103ee0 0x7efcdc1949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:25.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce20b1700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc0fee40 0x7efcdc19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:25.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce18b0700 1 -- 172.21.15.115:0/2875170951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc103ee0 msgr2=0x7efcdc1949f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:25.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce18b0700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc103ee0 0x7efcdc1949f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce18b0700 1 -- 172.21.15.115:0/2875170951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc0fee40 msgr2=0x7efcdc19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:25.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce18b0700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc0fee40 0x7efcdc19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce18b0700 1 -- 172.21.15.115:0/2875170951 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efccc0095f0 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce20b1700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc0fee40 0x7efcdc19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:25.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce18b0700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 0x7efcdc19aeb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efccc00b580 tx=0x7efccc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:25.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efcd37fe700 1 -- 172.21.15.115:0/2875170951 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc01d070 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efcd37fe700 1 -- 172.21.15.115:0/2875170951 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efccc003f40 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc195610 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efcd37fe700 1 -- 172.21.15.115:0/2875170951 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc022af0 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.251+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcdc1a1960 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.252+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcdc04e340 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.296 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.253+0000 7efcd37fe700 1 -- 172.21.15.115:0/2875170951 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7efccc0040b0 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.254+0000 7efcd37fe700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efcc80a1260 0x7efcc80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:25.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.254+0000 7efce20b1700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efcc80a1260 0x7efcc80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:25.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.254+0000 7efcd37fe700 1 -- 172.21.15.115:0/2875170951 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efccc0ce120 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.254+0000 7efce20b1700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efcc80a1260 0x7efcc80a3710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efcd800bc10 tx=0x7efcd800b5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:25.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.258+0000 7efcd37fe700 1 -- 172.21.15.115:0/2875170951 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efccc097550 con 0x7efcdc0ff7e0 2023-07-14T18:11:25.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.440+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efcdc0fcc00 con 0x7efcc80a1260 2023-07-14T18:11:25.491 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.452+0000 7efcd37fe700 1 -- 172.21.15.115:0/2875170951 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426832 (secure 0 0 0) 0x7efcdc0fcc00 con 0x7efcc80a1260 2023-07-14T18:11:25.496 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efcc80a1260 msgr2=0x7efcc80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efcc80a1260 0x7efcc80a3710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efcd800bc10 tx=0x7efcd800b5f0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 msgr2=0x7efcdc19aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 0x7efcdc19aeb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efccc00b580 tx=0x7efccc009580 comp rx=0 tx=0).stop 2023-07-14T18:11:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 shutdown_connections 2023-07-14T18:11:25.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7efcc80a1260 0x7efcc80a3710 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcdc103ee0 0x7efcdc1949f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcdc0fee40 0x7efcdc19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 --2- 172.21.15.115:0/2875170951 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcdc0ff7e0 0x7efcdc19aeb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 >> 172.21.15.115:0/2875170951 conn(0x7efcdc075ca0 msgr2=0x7efcdc0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 shutdown_connections 2023-07-14T18:11:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:25.455+0000 7efce4315700 1 -- 172.21.15.115:0/2875170951 wait complete. 2023-07-14T18:11:25.499 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2023-07-14T18:11:25.905 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2023-07-14T18:11:24.897605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653459,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":421,"num_read_kb":323,"num_write":337,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337592,"kb_used_data":13400,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604232,"statfs":{"total":1151940427776,"available":1151594733568,"internally_reserved":0,"allocated":13721600,"data_stored":7581014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003768"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089430+0000","last_change":"2023-07-14T18:10:18.110819+0000","last_active":"2023-07-14T18:10:24.089430+0000","last_peered":"2023-07-14T18:10:24.089430+0000","last_clean":"2023-07-14T18:10:24.089430+0000","last_became_active":"2023-07-14T18:10:18.110180+0000","last_became_peered":"2023-07-14T18:10:18.110180+0000","last_unstale":"2023-07-14T18:10:24.089430+0000","last_undegraded":"2023-07-14T18:10:24.089430+0000","last_fullsized":"2023-07-14T18:10:24.089430+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:08:50.579212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440800+0000","last_change":"2023-07-14T18:10:18.117539+0000","last_active":"2023-07-14T18:10:23.440800+0000","last_peered":"2023-07-14T18:10:23.440800+0000","last_clean":"2023-07-14T18:10:23.440800+0000","last_became_active":"2023-07-14T18:10:18.115956+0000","last_became_peered":"2023-07-14T18:10:18.115956+0000","last_unstale":"2023-07-14T18:10:23.440800+0000","last_undegraded":"2023-07-14T18:10:23.440800+0000","last_fullsized":"2023-07-14T18:10:23.440800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:22.723383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440427+0000","last_change":"2023-07-14T18:10:18.120582+0000","last_active":"2023-07-14T18:10:23.440427+0000","last_peered":"2023-07-14T18:10:23.440427+0000","last_clean":"2023-07-14T18:10:23.440427+0000","last_became_active":"2023-07-14T18:10:18.120296+0000","last_became_peered":"2023-07-14T18:10:18.120296+0000","last_unstale":"2023-07-14T18:10:23.440427+0000","last_undegraded":"2023-07-14T18:10:23.440427+0000","last_fullsized":"2023-07-14T18:10:23.440427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:54.381627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438612+0000","last_change":"2023-07-14T18:10:18.112250+0000","last_active":"2023-07-14T18:10:23.438612+0000","last_peered":"2023-07-14T18:10:23.438612+0000","last_clean":"2023-07-14T18:10:23.438612+0000","last_became_active":"2023-07-14T18:10:18.111514+0000","last_became_peered":"2023-07-14T18:10:18.111514+0000","last_unstale":"2023-07-14T18:10:23.438612+0000","last_undegraded":"2023-07-14T18:10:23.438612+0000","last_fullsized":"2023-07-14T18:10:23.438612+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:02:30.280559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438548+0000","last_change":"2023-07-14T18:10:18.111127+0000","last_active":"2023-07-14T18:10:23.438548+0000","last_peered":"2023-07-14T18:10:23.438548+0000","last_clean":"2023-07-14T18:10:23.438548+0000","last_became_active":"2023-07-14T18:10:18.110816+0000","last_became_peered":"2023-07-14T18:10:18.110816+0000","last_unstale":"2023-07-14T18:10:23.438548+0000","last_undegraded":"2023-07-14T18:10:23.438548+0000","last_fullsized":"2023-07-14T18:10:23.438548+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:05:59.027134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734705+0000","last_change":"2023-07-14T18:10:18.113436+0000","last_active":"2023-07-14T18:10:23.734705+0000","last_peered":"2023-07-14T18:10:23.734705+0000","last_clean":"2023-07-14T18:10:23.734705+0000","last_became_active":"2023-07-14T18:10:18.113172+0000","last_became_peered":"2023-07-14T18:10:18.113172+0000","last_unstale":"2023-07-14T18:10:23.734705+0000","last_undegraded":"2023-07-14T18:10:23.734705+0000","last_fullsized":"2023-07-14T18:10:23.734705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:33:57.446784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089034+0000","last_change":"2023-07-14T18:10:18.116720+0000","last_active":"2023-07-14T18:10:24.089034+0000","last_peered":"2023-07-14T18:10:24.089034+0000","last_clean":"2023-07-14T18:10:24.089034+0000","last_became_active":"2023-07-14T18:10:18.116361+0000","last_became_peered":"2023-07-14T18:10:18.116361+0000","last_unstale":"2023-07-14T18:10:24.089034+0000","last_undegraded":"2023-07-14T18:10:24.089034+0000","last_fullsized":"2023-07-14T18:10:24.089034+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:54.195458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088667+0000","last_change":"2023-07-14T18:10:18.111981+0000","last_active":"2023-07-14T18:10:24.088667+0000","last_peered":"2023-07-14T18:10:24.088667+0000","last_clean":"2023-07-14T18:10:24.088667+0000","last_became_active":"2023-07-14T18:10:18.111666+0000","last_became_peered":"2023-07-14T18:10:18.111666+0000","last_unstale":"2023-07-14T18:10:24.088667+0000","last_undegraded":"2023-07-14T18:10:24.088667+0000","last_fullsized":"2023-07-14T18:10:24.088667+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:07:35.710429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089557+0000","last_change":"2023-07-14T18:10:18.119497+0000","last_active":"2023-07-14T18:10:24.089557+0000","last_peered":"2023-07-14T18:10:24.089557+0000","last_clean":"2023-07-14T18:10:24.089557+0000","last_became_active":"2023-07-14T18:10:18.119090+0000","last_became_peered":"2023-07-14T18:10:18.119090+0000","last_unstale":"2023-07-14T18:10:24.089557+0000","last_undegraded":"2023-07-14T18:10:24.089557+0000","last_fullsized":"2023-07-14T18:10:24.089557+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:15:13.934598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.735011+0000","last_change":"2023-07-14T18:10:18.107353+0000","last_active":"2023-07-14T18:10:23.735011+0000","last_peered":"2023-07-14T18:10:23.735011+0000","last_clean":"2023-07-14T18:10:23.735011+0000","last_became_active":"2023-07-14T18:10:18.106933+0000","last_became_peered":"2023-07-14T18:10:18.106933+0000","last_unstale":"2023-07-14T18:10:23.735011+0000","last_undegraded":"2023-07-14T18:10:23.735011+0000","last_fullsized":"2023-07-14T18:10:23.735011+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:20:25.508991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089317+0000","last_change":"2023-07-14T18:10:18.121064+0000","last_active":"2023-07-14T18:10:24.089317+0000","last_peered":"2023-07-14T18:10:24.089317+0000","last_clean":"2023-07-14T18:10:24.089317+0000","last_became_active":"2023-07-14T18:10:18.120758+0000","last_became_peered":"2023-07-14T18:10:18.120758+0000","last_unstale":"2023-07-14T18:10:24.089317+0000","last_undegraded":"2023-07-14T18:10:24.089317+0000","last_fullsized":"2023-07-14T18:10:24.089317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:42.972896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438568+0000","last_change":"2023-07-14T18:10:18.112436+0000","last_active":"2023-07-14T18:10:23.438568+0000","last_peered":"2023-07-14T18:10:23.438568+0000","last_clean":"2023-07-14T18:10:23.438568+0000","last_became_active":"2023-07-14T18:10:18.111942+0000","last_became_peered":"2023-07-14T18:10:18.111942+0000","last_unstale":"2023-07-14T18:10:23.438568+0000","last_undegraded":"2023-07-14T18:10:23.438568+0000","last_fullsized":"2023-07-14T18:10:23.438568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:57.856511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845884+0000","last_change":"2023-07-14T18:10:18.106134+0000","last_active":"2023-07-14T18:10:23.845884+0000","last_peered":"2023-07-14T18:10:23.845884+0000","last_clean":"2023-07-14T18:10:23.845884+0000","last_became_active":"2023-07-14T18:10:18.105481+0000","last_became_peered":"2023-07-14T18:10:18.105481+0000","last_unstale":"2023-07-14T18:10:23.845884+0000","last_undegraded":"2023-07-14T18:10:23.845884+0000","last_fullsized":"2023-07-14T18:10:23.845884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:53.958762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440197+0000","last_change":"2023-07-14T18:10:18.117399+0000","last_active":"2023-07-14T18:10:23.440197+0000","last_peered":"2023-07-14T18:10:23.440197+0000","last_clean":"2023-07-14T18:10:23.440197+0000","last_became_active":"2023-07-14T18:10:18.117094+0000","last_became_peered":"2023-07-14T18:10:18.117094+0000","last_unstale":"2023-07-14T18:10:23.440197+0000","last_undegraded":"2023-07-14T18:10:23.440197+0000","last_fullsized":"2023-07-14T18:10:23.440197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:10:09.445305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440157+0000","last_change":"2023-07-14T18:10:18.117150+0000","last_active":"2023-07-14T18:10:23.440157+0000","last_peered":"2023-07-14T18:10:23.440157+0000","last_clean":"2023-07-14T18:10:23.440157+0000","last_became_active":"2023-07-14T18:10:18.116805+0000","last_became_peered":"2023-07-14T18:10:18.116805+0000","last_unstale":"2023-07-14T18:10:23.440157+0000","last_undegraded":"2023-07-14T18:10:23.440157+0000","last_fullsized":"2023-07-14T18:10:23.440157+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:48:43.803465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088549+0000","last_change":"2023-07-14T18:10:18.120868+0000","last_active":"2023-07-14T18:10:24.088549+0000","last_peered":"2023-07-14T18:10:24.088549+0000","last_clean":"2023-07-14T18:10:24.088549+0000","last_became_active":"2023-07-14T18:10:18.120083+0000","last_became_peered":"2023-07-14T18:10:18.120083+0000","last_unstale":"2023-07-14T18:10:24.088549+0000","last_undegraded":"2023-07-14T18:10:24.088549+0000","last_fullsized":"2023-07-14T18:10:24.088549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:50:07.162775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089298+0000","last_change":"2023-07-14T18:10:18.111011+0000","last_active":"2023-07-14T18:10:24.089298+0000","last_peered":"2023-07-14T18:10:24.089298+0000","last_clean":"2023-07-14T18:10:24.089298+0000","last_became_active":"2023-07-14T18:10:18.110517+0000","last_became_peered":"2023-07-14T18:10:18.110517+0000","last_unstale":"2023-07-14T18:10:24.089298+0000","last_undegraded":"2023-07-14T18:10:24.089298+0000","last_fullsized":"2023-07-14T18:10:24.089298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:00:45.278711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089798+0000","last_change":"2023-07-14T18:10:18.110739+0000","last_active":"2023-07-14T18:10:24.089798+0000","last_peered":"2023-07-14T18:10:24.089798+0000","last_clean":"2023-07-14T18:10:24.089798+0000","last_became_active":"2023-07-14T18:10:18.110442+0000","last_became_peered":"2023-07-14T18:10:18.110442+0000","last_unstale":"2023-07-14T18:10:24.089798+0000","last_undegraded":"2023-07-14T18:10:24.089798+0000","last_fullsized":"2023-07-14T18:10:24.089798+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:45.130540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550066+0000","last_change":"2023-07-14T18:10:18.109955+0000","last_active":"2023-07-14T18:10:23.550066+0000","last_peered":"2023-07-14T18:10:23.550066+0000","last_clean":"2023-07-14T18:10:23.550066+0000","last_became_active":"2023-07-14T18:10:18.109651+0000","last_became_peered":"2023-07-14T18:10:18.109651+0000","last_unstale":"2023-07-14T18:10:23.550066+0000","last_undegraded":"2023-07-14T18:10:23.550066+0000","last_fullsized":"2023-07-14T18:10:23.550066+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:22:46.479116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549977+0000","last_change":"2023-07-14T18:10:22.457996+0000","last_active":"2023-07-14T18:10:23.549977+0000","last_peered":"2023-07-14T18:10:23.549977+0000","last_clean":"2023-07-14T18:10:23.549977+0000","last_became_active":"2023-07-14T18:10:22.457573+0000","last_became_peered":"2023-07-14T18:10:22.457573+0000","last_unstale":"2023-07-14T18:10:23.549977+0000","last_undegraded":"2023-07-14T18:10:23.549977+0000","last_fullsized":"2023-07-14T18:10:23.549977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:17.148116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429932+0000","last_change":"2023-07-14T18:10:17.216780+0000","last_active":"2023-07-14T18:10:37.429932+0000","last_peered":"2023-07-14T18:10:37.429932+0000","last_clean":"2023-07-14T18:10:37.429932+0000","last_became_active":"2023-07-14T18:10:17.216423+0000","last_became_peered":"2023-07-14T18:10:17.216423+0000","last_unstale":"2023-07-14T18:10:37.429932+0000","last_undegraded":"2023-07-14T18:10:37.429932+0000","last_fullsized":"2023-07-14T18:10:37.429932+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:24.522131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089127+0000","last_change":"2023-07-14T18:10:18.108615+0000","last_active":"2023-07-14T18:10:24.089127+0000","last_peered":"2023-07-14T18:10:24.089127+0000","last_clean":"2023-07-14T18:10:24.089127+0000","last_became_active":"2023-07-14T18:10:18.108316+0000","last_became_peered":"2023-07-14T18:10:18.108316+0000","last_unstale":"2023-07-14T18:10:24.089127+0000","last_undegraded":"2023-07-14T18:10:24.089127+0000","last_fullsized":"2023-07-14T18:10:24.089127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:27:25.084342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438203+0000","last_change":"2023-07-14T18:10:22.453040+0000","last_active":"2023-07-14T18:10:23.438203+0000","last_peered":"2023-07-14T18:10:23.438203+0000","last_clean":"2023-07-14T18:10:23.438203+0000","last_became_active":"2023-07-14T18:10:22.452643+0000","last_became_peered":"2023-07-14T18:10:22.452643+0000","last_unstale":"2023-07-14T18:10:23.438203+0000","last_undegraded":"2023-07-14T18:10:23.438203+0000","last_fullsized":"2023-07-14T18:10:23.438203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:18.191433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089445+0000","last_change":"2023-07-14T18:10:17.214085+0000","last_active":"2023-07-14T18:10:24.089445+0000","last_peered":"2023-07-14T18:10:24.089445+0000","last_clean":"2023-07-14T18:10:24.089445+0000","last_became_active":"2023-07-14T18:10:17.213772+0000","last_became_peered":"2023-07-14T18:10:17.213772+0000","last_unstale":"2023-07-14T18:10:24.089445+0000","last_undegraded":"2023-07-14T18:10:24.089445+0000","last_fullsized":"2023-07-14T18:10:24.089445+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:16:35.586284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440705+0000","last_change":"2023-07-14T18:10:18.120532+0000","last_active":"2023-07-14T18:10:23.440705+0000","last_peered":"2023-07-14T18:10:23.440705+0000","last_clean":"2023-07-14T18:10:23.440705+0000","last_became_active":"2023-07-14T18:10:18.120199+0000","last_became_peered":"2023-07-14T18:10:18.120199+0000","last_unstale":"2023-07-14T18:10:23.440705+0000","last_undegraded":"2023-07-14T18:10:23.440705+0000","last_fullsized":"2023-07-14T18:10:23.440705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:05:38.292570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438439+0000","last_change":"2023-07-14T18:10:22.458820+0000","last_active":"2023-07-14T18:10:23.438439+0000","last_peered":"2023-07-14T18:10:23.438439+0000","last_clean":"2023-07-14T18:10:23.438439+0000","last_became_active":"2023-07-14T18:10:22.458480+0000","last_became_peered":"2023-07-14T18:10:22.458480+0000","last_unstale":"2023-07-14T18:10:23.438439+0000","last_undegraded":"2023-07-14T18:10:23.438439+0000","last_fullsized":"2023-07-14T18:10:23.438439+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:45.901536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428015+0000","last_change":"2023-07-14T18:10:17.212329+0000","last_active":"2023-07-14T18:10:37.428015+0000","last_peered":"2023-07-14T18:10:37.428015+0000","last_clean":"2023-07-14T18:10:37.428015+0000","last_became_active":"2023-07-14T18:10:17.212013+0000","last_became_peered":"2023-07-14T18:10:17.212013+0000","last_unstale":"2023-07-14T18:10:37.428015+0000","last_undegraded":"2023-07-14T18:10:37.428015+0000","last_fullsized":"2023-07-14T18:10:37.428015+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:19:15.950647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089229+0000","last_change":"2023-07-14T18:10:18.114621+0000","last_active":"2023-07-14T18:10:24.089229+0000","last_peered":"2023-07-14T18:10:24.089229+0000","last_clean":"2023-07-14T18:10:24.089229+0000","last_became_active":"2023-07-14T18:10:18.114330+0000","last_became_peered":"2023-07-14T18:10:18.114330+0000","last_unstale":"2023-07-14T18:10:24.089229+0000","last_undegraded":"2023-07-14T18:10:24.089229+0000","last_fullsized":"2023-07-14T18:10:24.089229+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:47:21.060539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438244+0000","last_change":"2023-07-14T18:10:22.453179+0000","last_active":"2023-07-14T18:10:23.438244+0000","last_peered":"2023-07-14T18:10:23.438244+0000","last_clean":"2023-07-14T18:10:23.438244+0000","last_became_active":"2023-07-14T18:10:22.452745+0000","last_became_peered":"2023-07-14T18:10:22.452745+0000","last_unstale":"2023-07-14T18:10:23.438244+0000","last_undegraded":"2023-07-14T18:10:23.438244+0000","last_fullsized":"2023-07-14T18:10:23.438244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:22:16.784956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438868+0000","last_change":"2023-07-14T18:10:17.215115+0000","last_active":"2023-07-14T18:10:23.438868+0000","last_peered":"2023-07-14T18:10:23.438868+0000","last_clean":"2023-07-14T18:10:23.438868+0000","last_became_active":"2023-07-14T18:10:17.214813+0000","last_became_peered":"2023-07-14T18:10:17.214813+0000","last_unstale":"2023-07-14T18:10:23.438868+0000","last_undegraded":"2023-07-14T18:10:23.438868+0000","last_fullsized":"2023-07-14T18:10:23.438868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:48:32.267436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090280+0000","last_change":"2023-07-14T18:10:18.119987+0000","last_active":"2023-07-14T18:10:24.090280+0000","last_peered":"2023-07-14T18:10:24.090280+0000","last_clean":"2023-07-14T18:10:24.090280+0000","last_became_active":"2023-07-14T18:10:18.119695+0000","last_became_peered":"2023-07-14T18:10:18.119695+0000","last_unstale":"2023-07-14T18:10:24.090280+0000","last_undegraded":"2023-07-14T18:10:24.090280+0000","last_fullsized":"2023-07-14T18:10:24.090280+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:46.295719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550007+0000","last_change":"2023-07-14T18:10:22.449689+0000","last_active":"2023-07-14T18:10:23.550007+0000","last_peered":"2023-07-14T18:10:23.550007+0000","last_clean":"2023-07-14T18:10:23.550007+0000","last_became_active":"2023-07-14T18:10:22.449273+0000","last_became_peered":"2023-07-14T18:10:22.449273+0000","last_unstale":"2023-07-14T18:10:23.550007+0000","last_undegraded":"2023-07-14T18:10:23.550007+0000","last_fullsized":"2023-07-14T18:10:23.550007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:54:23.642287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:20.487923+0000","last_change":"2023-07-14T18:10:17.216438+0000","last_active":"2023-07-14T18:11:20.487923+0000","last_peered":"2023-07-14T18:11:20.487923+0000","last_clean":"2023-07-14T18:11:20.487923+0000","last_became_active":"2023-07-14T18:10:17.216169+0000","last_became_peered":"2023-07-14T18:10:17.216169+0000","last_unstale":"2023-07-14T18:11:20.487923+0000","last_undegraded":"2023-07-14T18:11:20.487923+0000","last_fullsized":"2023-07-14T18:11:20.487923+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:39.977001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438762+0000","last_change":"2023-07-14T18:10:18.112917+0000","last_active":"2023-07-14T18:10:23.438762+0000","last_peered":"2023-07-14T18:10:23.438762+0000","last_clean":"2023-07-14T18:10:23.438762+0000","last_became_active":"2023-07-14T18:10:18.112631+0000","last_became_peered":"2023-07-14T18:10:18.112631+0000","last_unstale":"2023-07-14T18:10:23.438762+0000","last_undegraded":"2023-07-14T18:10:23.438762+0000","last_fullsized":"2023-07-14T18:10:23.438762+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:14:01.774309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549936+0000","last_change":"2023-07-14T18:10:22.454278+0000","last_active":"2023-07-14T18:10:23.549936+0000","last_peered":"2023-07-14T18:10:23.549936+0000","last_clean":"2023-07-14T18:10:23.549936+0000","last_became_active":"2023-07-14T18:10:22.453834+0000","last_became_peered":"2023-07-14T18:10:22.453834+0000","last_unstale":"2023-07-14T18:10:23.549936+0000","last_undegraded":"2023-07-14T18:10:23.549936+0000","last_fullsized":"2023-07-14T18:10:23.549936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:31:24.983615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430580+0000","last_change":"2023-07-14T18:10:17.210217+0000","last_active":"2023-07-14T18:10:37.430580+0000","last_peered":"2023-07-14T18:10:37.430580+0000","last_clean":"2023-07-14T18:10:37.430580+0000","last_became_active":"2023-07-14T18:10:17.209953+0000","last_became_peered":"2023-07-14T18:10:17.209953+0000","last_unstale":"2023-07-14T18:10:37.430580+0000","last_undegraded":"2023-07-14T18:10:37.430580+0000","last_fullsized":"2023-07-14T18:10:37.430580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:19:46.091265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090402+0000","last_change":"2023-07-14T18:10:18.113009+0000","last_active":"2023-07-14T18:10:24.090402+0000","last_peered":"2023-07-14T18:10:24.090402+0000","last_clean":"2023-07-14T18:10:24.090402+0000","last_became_active":"2023-07-14T18:10:18.112695+0000","last_became_peered":"2023-07-14T18:10:18.112695+0000","last_unstale":"2023-07-14T18:10:24.090402+0000","last_undegraded":"2023-07-14T18:10:24.090402+0000","last_fullsized":"2023-07-14T18:10:24.090402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:06:04.361736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089159+0000","last_change":"2023-07-14T18:10:22.457687+0000","last_active":"2023-07-14T18:10:24.089159+0000","last_peered":"2023-07-14T18:10:24.089159+0000","last_clean":"2023-07-14T18:10:24.089159+0000","last_became_active":"2023-07-14T18:10:22.457270+0000","last_became_peered":"2023-07-14T18:10:22.457270+0000","last_unstale":"2023-07-14T18:10:24.089159+0000","last_undegraded":"2023-07-14T18:10:24.089159+0000","last_fullsized":"2023-07-14T18:10:24.089159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:37:58.192527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429469+0000","last_change":"2023-07-14T18:10:17.209638+0000","last_active":"2023-07-14T18:10:37.429469+0000","last_peered":"2023-07-14T18:10:37.429469+0000","last_clean":"2023-07-14T18:10:37.429469+0000","last_became_active":"2023-07-14T18:10:17.209409+0000","last_became_peered":"2023-07-14T18:10:17.209409+0000","last_unstale":"2023-07-14T18:10:37.429469+0000","last_undegraded":"2023-07-14T18:10:37.429469+0000","last_fullsized":"2023-07-14T18:10:37.429469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:32.419102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088533+0000","last_change":"2023-07-14T18:10:18.109479+0000","last_active":"2023-07-14T18:10:24.088533+0000","last_peered":"2023-07-14T18:10:24.088533+0000","last_clean":"2023-07-14T18:10:24.088533+0000","last_became_active":"2023-07-14T18:10:18.109161+0000","last_became_peered":"2023-07-14T18:10:18.109161+0000","last_unstale":"2023-07-14T18:10:24.088533+0000","last_undegraded":"2023-07-14T18:10:24.088533+0000","last_fullsized":"2023-07-14T18:10:24.088533+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:05.204295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440596+0000","last_change":"2023-07-14T18:10:22.453731+0000","last_active":"2023-07-14T18:10:23.440596+0000","last_peered":"2023-07-14T18:10:23.440596+0000","last_clean":"2023-07-14T18:10:23.440596+0000","last_became_active":"2023-07-14T18:10:22.453311+0000","last_became_peered":"2023-07-14T18:10:22.453311+0000","last_unstale":"2023-07-14T18:10:23.440596+0000","last_undegraded":"2023-07-14T18:10:23.440596+0000","last_fullsized":"2023-07-14T18:10:23.440596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:35.892698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089898+0000","last_change":"2023-07-14T18:10:17.218050+0000","last_active":"2023-07-14T18:10:24.089898+0000","last_peered":"2023-07-14T18:10:24.089898+0000","last_clean":"2023-07-14T18:10:24.089898+0000","last_became_active":"2023-07-14T18:10:17.217793+0000","last_became_peered":"2023-07-14T18:10:17.217793+0000","last_unstale":"2023-07-14T18:10:24.089898+0000","last_undegraded":"2023-07-14T18:10:24.089898+0000","last_fullsized":"2023-07-14T18:10:24.089898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:23.781098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089467+0000","last_change":"2023-07-14T18:10:18.111598+0000","last_active":"2023-07-14T18:10:24.089467+0000","last_peered":"2023-07-14T18:10:24.089467+0000","last_clean":"2023-07-14T18:10:24.089467+0000","last_became_active":"2023-07-14T18:10:18.111244+0000","last_became_peered":"2023-07-14T18:10:18.111244+0000","last_unstale":"2023-07-14T18:10:24.089467+0000","last_undegraded":"2023-07-14T18:10:24.089467+0000","last_fullsized":"2023-07-14T18:10:24.089467+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:26:47.187587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088793+0000","last_change":"2023-07-14T18:10:22.447676+0000","last_active":"2023-07-14T18:10:24.088793+0000","last_peered":"2023-07-14T18:10:24.088793+0000","last_clean":"2023-07-14T18:10:24.088793+0000","last_became_active":"2023-07-14T18:10:22.446648+0000","last_became_peered":"2023-07-14T18:10:22.446648+0000","last_unstale":"2023-07-14T18:10:24.088793+0000","last_undegraded":"2023-07-14T18:10:24.088793+0000","last_fullsized":"2023-07-14T18:10:24.088793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:17:30.941899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:19.486536+0000","last_change":"2023-07-14T18:10:17.213897+0000","last_active":"2023-07-14T18:11:19.486536+0000","last_peered":"2023-07-14T18:11:19.486536+0000","last_clean":"2023-07-14T18:11:19.486536+0000","last_became_active":"2023-07-14T18:10:17.213547+0000","last_became_peered":"2023-07-14T18:10:17.213547+0000","last_unstale":"2023-07-14T18:11:19.486536+0000","last_undegraded":"2023-07-14T18:11:19.486536+0000","last_fullsized":"2023-07-14T18:11:19.486536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:44.891524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089327+0000","last_change":"2023-07-14T18:10:18.114776+0000","last_active":"2023-07-14T18:10:24.089327+0000","last_peered":"2023-07-14T18:10:24.089327+0000","last_clean":"2023-07-14T18:10:24.089327+0000","last_became_active":"2023-07-14T18:10:18.112469+0000","last_became_peered":"2023-07-14T18:10:18.112469+0000","last_unstale":"2023-07-14T18:10:24.089327+0000","last_undegraded":"2023-07-14T18:10:24.089327+0000","last_fullsized":"2023-07-14T18:10:24.089327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:30.864496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088873+0000","last_change":"2023-07-14T18:10:22.454215+0000","last_active":"2023-07-14T18:10:24.088873+0000","last_peered":"2023-07-14T18:10:24.088873+0000","last_clean":"2023-07-14T18:10:24.088873+0000","last_became_active":"2023-07-14T18:10:22.453889+0000","last_became_peered":"2023-07-14T18:10:22.453889+0000","last_unstale":"2023-07-14T18:10:24.088873+0000","last_undegraded":"2023-07-14T18:10:24.088873+0000","last_fullsized":"2023-07-14T18:10:24.088873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:30:09.159482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981368+0000","last_change":"2023-07-14T18:10:17.212602+0000","last_active":"2023-07-14T18:10:36.981368+0000","last_peered":"2023-07-14T18:10:36.981368+0000","last_clean":"2023-07-14T18:10:36.981368+0000","last_became_active":"2023-07-14T18:10:17.212087+0000","last_became_peered":"2023-07-14T18:10:17.212087+0000","last_unstale":"2023-07-14T18:10:36.981368+0000","last_undegraded":"2023-07-14T18:10:36.981368+0000","last_fullsized":"2023-07-14T18:10:36.981368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:30:00.562894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440688+0000","last_change":"2023-07-14T18:10:18.116972+0000","last_active":"2023-07-14T18:10:23.440688+0000","last_peered":"2023-07-14T18:10:23.440688+0000","last_clean":"2023-07-14T18:10:23.440688+0000","last_became_active":"2023-07-14T18:10:18.116423+0000","last_became_peered":"2023-07-14T18:10:18.116423+0000","last_unstale":"2023-07-14T18:10:23.440688+0000","last_undegraded":"2023-07-14T18:10:23.440688+0000","last_fullsized":"2023-07-14T18:10:23.440688+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:53.315535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438563+0000","last_change":"2023-07-14T18:10:22.450207+0000","last_active":"2023-07-14T18:10:23.438563+0000","last_peered":"2023-07-14T18:10:23.438563+0000","last_clean":"2023-07-14T18:10:23.438563+0000","last_became_active":"2023-07-14T18:10:22.449826+0000","last_became_peered":"2023-07-14T18:10:22.449826+0000","last_unstale":"2023-07-14T18:10:23.438563+0000","last_undegraded":"2023-07-14T18:10:23.438563+0000","last_fullsized":"2023-07-14T18:10:23.438563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:18.013945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.982272+0000","last_change":"2023-07-14T18:10:17.213769+0000","last_active":"2023-07-14T18:10:36.982272+0000","last_peered":"2023-07-14T18:10:36.982272+0000","last_clean":"2023-07-14T18:10:36.982272+0000","last_became_active":"2023-07-14T18:10:17.213465+0000","last_became_peered":"2023-07-14T18:10:17.213465+0000","last_unstale":"2023-07-14T18:10:36.982272+0000","last_undegraded":"2023-07-14T18:10:36.982272+0000","last_fullsized":"2023-07-14T18:10:36.982272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:38:07.890716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089153+0000","last_change":"2023-07-14T18:10:18.122764+0000","last_active":"2023-07-14T18:10:24.089153+0000","last_peered":"2023-07-14T18:10:24.089153+0000","last_clean":"2023-07-14T18:10:24.089153+0000","last_became_active":"2023-07-14T18:10:18.120522+0000","last_became_peered":"2023-07-14T18:10:18.120522+0000","last_unstale":"2023-07-14T18:10:24.089153+0000","last_undegraded":"2023-07-14T18:10:24.089153+0000","last_fullsized":"2023-07-14T18:10:24.089153+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:54:07.567008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089725+0000","last_change":"2023-07-14T18:10:22.455635+0000","last_active":"2023-07-14T18:10:24.089725+0000","last_peered":"2023-07-14T18:10:24.089725+0000","last_clean":"2023-07-14T18:10:24.089725+0000","last_became_active":"2023-07-14T18:10:22.455128+0000","last_became_peered":"2023-07-14T18:10:22.455128+0000","last_unstale":"2023-07-14T18:10:24.089725+0000","last_undegraded":"2023-07-14T18:10:24.089725+0000","last_fullsized":"2023-07-14T18:10:24.089725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:01.364723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'31","reported_seq":149,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:20.487400+0000","last_change":"2023-07-14T18:10:17.215255+0000","last_active":"2023-07-14T18:11:20.487400+0000","last_peered":"2023-07-14T18:11:20.487400+0000","last_clean":"2023-07-14T18:11:20.487400+0000","last_became_active":"2023-07-14T18:10:17.214900+0000","last_became_peered":"2023-07-14T18:10:17.214900+0000","last_unstale":"2023-07-14T18:11:20.487400+0000","last_undegraded":"2023-07-14T18:11:20.487400+0000","last_fullsized":"2023-07-14T18:11:20.487400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:01.480644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":96,"num_read_kb":104,"num_write":32,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438902+0000","last_change":"2023-07-14T18:10:18.115355+0000","last_active":"2023-07-14T18:10:23.438902+0000","last_peered":"2023-07-14T18:10:23.438902+0000","last_clean":"2023-07-14T18:10:23.438902+0000","last_became_active":"2023-07-14T18:10:18.115023+0000","last_became_peered":"2023-07-14T18:10:18.115023+0000","last_unstale":"2023-07-14T18:10:23.438902+0000","last_undegraded":"2023-07-14T18:10:23.438902+0000","last_fullsized":"2023-07-14T18:10:23.438902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:16.057123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090803+0000","last_change":"2023-07-14T18:10:22.453113+0000","last_active":"2023-07-14T18:10:24.090803+0000","last_peered":"2023-07-14T18:10:24.090803+0000","last_clean":"2023-07-14T18:10:24.090803+0000","last_became_active":"2023-07-14T18:10:22.452688+0000","last_became_peered":"2023-07-14T18:10:22.452688+0000","last_unstale":"2023-07-14T18:10:24.090803+0000","last_undegraded":"2023-07-14T18:10:24.090803+0000","last_fullsized":"2023-07-14T18:10:24.090803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:47.829022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430020+0000","last_change":"2023-07-14T18:10:17.210924+0000","last_active":"2023-07-14T18:10:37.430020+0000","last_peered":"2023-07-14T18:10:37.430020+0000","last_clean":"2023-07-14T18:10:37.430020+0000","last_became_active":"2023-07-14T18:10:17.210604+0000","last_became_peered":"2023-07-14T18:10:17.210604+0000","last_unstale":"2023-07-14T18:10:37.430020+0000","last_undegraded":"2023-07-14T18:10:37.430020+0000","last_fullsized":"2023-07-14T18:10:37.430020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:10:28.635558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845862+0000","last_change":"2023-07-14T18:10:18.115094+0000","last_active":"2023-07-14T18:10:23.845862+0000","last_peered":"2023-07-14T18:10:23.845862+0000","last_clean":"2023-07-14T18:10:23.845862+0000","last_became_active":"2023-07-14T18:10:18.114538+0000","last_became_peered":"2023-07-14T18:10:18.114538+0000","last_unstale":"2023-07-14T18:10:23.845862+0000","last_undegraded":"2023-07-14T18:10:23.845862+0000","last_fullsized":"2023-07-14T18:10:23.845862+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:19:15.916223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088869+0000","last_change":"2023-07-14T18:10:22.459151+0000","last_active":"2023-07-14T18:10:24.088869+0000","last_peered":"2023-07-14T18:10:24.088869+0000","last_clean":"2023-07-14T18:10:24.088869+0000","last_became_active":"2023-07-14T18:10:22.458550+0000","last_became_peered":"2023-07-14T18:10:22.458550+0000","last_unstale":"2023-07-14T18:10:24.088869+0000","last_undegraded":"2023-07-14T18:10:24.088869+0000","last_fullsized":"2023-07-14T18:10:24.088869+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:11.750342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.982150+0000","last_change":"2023-07-14T18:10:17.216475+0000","last_active":"2023-07-14T18:10:36.982150+0000","last_peered":"2023-07-14T18:10:36.982150+0000","last_clean":"2023-07-14T18:10:36.982150+0000","last_became_active":"2023-07-14T18:10:17.216148+0000","last_became_peered":"2023-07-14T18:10:17.216148+0000","last_unstale":"2023-07-14T18:10:36.982150+0000","last_undegraded":"2023-07-14T18:10:36.982150+0000","last_fullsized":"2023-07-14T18:10:36.982150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:00:50.106730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089545+0000","last_change":"2023-07-14T18:10:18.121764+0000","last_active":"2023-07-14T18:10:24.089545+0000","last_peered":"2023-07-14T18:10:24.089545+0000","last_clean":"2023-07-14T18:10:24.089545+0000","last_became_active":"2023-07-14T18:10:18.121458+0000","last_became_peered":"2023-07-14T18:10:18.121458+0000","last_unstale":"2023-07-14T18:10:24.089545+0000","last_undegraded":"2023-07-14T18:10:24.089545+0000","last_fullsized":"2023-07-14T18:10:24.089545+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:26:22.145994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089867+0000","last_change":"2023-07-14T18:10:22.457737+0000","last_active":"2023-07-14T18:10:24.089867+0000","last_peered":"2023-07-14T18:10:24.089867+0000","last_clean":"2023-07-14T18:10:24.089867+0000","last_became_active":"2023-07-14T18:10:22.457359+0000","last_became_peered":"2023-07-14T18:10:22.457359+0000","last_unstale":"2023-07-14T18:10:24.089867+0000","last_undegraded":"2023-07-14T18:10:24.089867+0000","last_fullsized":"2023-07-14T18:10:24.089867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:55.574003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'4","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:20.500023+0000","last_change":"2023-07-14T18:10:17.214663+0000","last_active":"2023-07-14T18:11:20.500023+0000","last_peered":"2023-07-14T18:11:20.500023+0000","last_clean":"2023-07-14T18:11:20.500023+0000","last_became_active":"2023-07-14T18:10:17.214355+0000","last_became_peered":"2023-07-14T18:10:17.214355+0000","last_unstale":"2023-07-14T18:11:20.500023+0000","last_undegraded":"2023-07-14T18:11:20.500023+0000","last_fullsized":"2023-07-14T18:11:20.500023+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:56.306784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088750+0000","last_change":"2023-07-14T18:10:18.124903+0000","last_active":"2023-07-14T18:10:24.088750+0000","last_peered":"2023-07-14T18:10:24.088750+0000","last_clean":"2023-07-14T18:10:24.088750+0000","last_became_active":"2023-07-14T18:10:18.124661+0000","last_became_peered":"2023-07-14T18:10:18.124661+0000","last_unstale":"2023-07-14T18:10:24.088750+0000","last_undegraded":"2023-07-14T18:10:24.088750+0000","last_fullsized":"2023-07-14T18:10:24.088750+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:21.959726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846209+0000","last_change":"2023-07-14T18:10:22.459746+0000","last_active":"2023-07-14T18:10:23.846209+0000","last_peered":"2023-07-14T18:10:23.846209+0000","last_clean":"2023-07-14T18:10:23.846209+0000","last_became_active":"2023-07-14T18:10:22.459331+0000","last_became_peered":"2023-07-14T18:10:22.459331+0000","last_unstale":"2023-07-14T18:10:23.846209+0000","last_undegraded":"2023-07-14T18:10:23.846209+0000","last_fullsized":"2023-07-14T18:10:23.846209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:34.082947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.430317+0000","last_change":"2023-07-14T18:10:17.213362+0000","last_active":"2023-07-14T18:10:37.430317+0000","last_peered":"2023-07-14T18:10:37.430317+0000","last_clean":"2023-07-14T18:10:37.430317+0000","last_became_active":"2023-07-14T18:10:17.212914+0000","last_became_peered":"2023-07-14T18:10:17.212914+0000","last_unstale":"2023-07-14T18:10:37.430317+0000","last_undegraded":"2023-07-14T18:10:37.430317+0000","last_fullsized":"2023-07-14T18:10:37.430317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:08.051946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090127+0000","last_change":"2023-07-14T18:10:18.106583+0000","last_active":"2023-07-14T18:10:24.090127+0000","last_peered":"2023-07-14T18:10:24.090127+0000","last_clean":"2023-07-14T18:10:24.090127+0000","last_became_active":"2023-07-14T18:10:18.106216+0000","last_became_peered":"2023-07-14T18:10:18.106216+0000","last_unstale":"2023-07-14T18:10:24.090127+0000","last_undegraded":"2023-07-14T18:10:24.090127+0000","last_fullsized":"2023-07-14T18:10:24.090127+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:03:32.095853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089940+0000","last_change":"2023-07-14T18:10:22.447133+0000","last_active":"2023-07-14T18:10:24.089940+0000","last_peered":"2023-07-14T18:10:24.089940+0000","last_clean":"2023-07-14T18:10:24.089940+0000","last_became_active":"2023-07-14T18:10:22.446716+0000","last_became_peered":"2023-07-14T18:10:22.446716+0000","last_unstale":"2023-07-14T18:10:24.089940+0000","last_undegraded":"2023-07-14T18:10:24.089940+0000","last_fullsized":"2023-07-14T18:10:24.089940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:49.677906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.419316+0000","last_change":"2023-07-14T18:10:17.210729+0000","last_active":"2023-07-14T18:10:37.419316+0000","last_peered":"2023-07-14T18:10:37.419316+0000","last_clean":"2023-07-14T18:10:37.419316+0000","last_became_active":"2023-07-14T18:10:17.210449+0000","last_became_peered":"2023-07-14T18:10:17.210449+0000","last_unstale":"2023-07-14T18:10:37.419316+0000","last_undegraded":"2023-07-14T18:10:37.419316+0000","last_fullsized":"2023-07-14T18:10:37.419316+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:57.909641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089476+0000","last_change":"2023-07-14T18:10:18.107118+0000","last_active":"2023-07-14T18:10:24.089476+0000","last_peered":"2023-07-14T18:10:24.089476+0000","last_clean":"2023-07-14T18:10:24.089476+0000","last_became_active":"2023-07-14T18:10:18.106367+0000","last_became_peered":"2023-07-14T18:10:18.106367+0000","last_unstale":"2023-07-14T18:10:24.089476+0000","last_undegraded":"2023-07-14T18:10:24.089476+0000","last_fullsized":"2023-07-14T18:10:24.089476+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:50:19.299272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438592+0000","last_change":"2023-07-14T18:10:22.500692+0000","last_active":"2023-07-14T18:10:23.438592+0000","last_peered":"2023-07-14T18:10:23.438592+0000","last_clean":"2023-07-14T18:10:23.438592+0000","last_became_active":"2023-07-14T18:10:22.500239+0000","last_became_peered":"2023-07-14T18:10:22.500239+0000","last_unstale":"2023-07-14T18:10:23.438592+0000","last_undegraded":"2023-07-14T18:10:23.438592+0000","last_fullsized":"2023-07-14T18:10:23.438592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:06.716562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089450+0000","last_change":"2023-07-14T18:10:17.212519+0000","last_active":"2023-07-14T18:10:24.089450+0000","last_peered":"2023-07-14T18:10:24.089450+0000","last_clean":"2023-07-14T18:10:24.089450+0000","last_became_active":"2023-07-14T18:10:17.212228+0000","last_became_peered":"2023-07-14T18:10:17.212228+0000","last_unstale":"2023-07-14T18:10:24.089450+0000","last_undegraded":"2023-07-14T18:10:24.089450+0000","last_fullsized":"2023-07-14T18:10:24.089450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:25:19.995310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734905+0000","last_change":"2023-07-14T18:10:18.111808+0000","last_active":"2023-07-14T18:10:23.734905+0000","last_peered":"2023-07-14T18:10:23.734905+0000","last_clean":"2023-07-14T18:10:23.734905+0000","last_became_active":"2023-07-14T18:10:18.111501+0000","last_became_peered":"2023-07-14T18:10:18.111501+0000","last_unstale":"2023-07-14T18:10:23.734905+0000","last_undegraded":"2023-07-14T18:10:23.734905+0000","last_fullsized":"2023-07-14T18:10:23.734905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:36.111363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845939+0000","last_change":"2023-07-14T18:10:22.453153+0000","last_active":"2023-07-14T18:10:23.845939+0000","last_peered":"2023-07-14T18:10:23.845939+0000","last_clean":"2023-07-14T18:10:23.845939+0000","last_became_active":"2023-07-14T18:10:22.452767+0000","last_became_peered":"2023-07-14T18:10:22.452767+0000","last_unstale":"2023-07-14T18:10:23.845939+0000","last_undegraded":"2023-07-14T18:10:23.845939+0000","last_fullsized":"2023-07-14T18:10:23.845939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:10:07.917749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089769+0000","last_change":"2023-07-14T18:10:17.215950+0000","last_active":"2023-07-14T18:10:24.089769+0000","last_peered":"2023-07-14T18:10:24.089769+0000","last_clean":"2023-07-14T18:10:24.089769+0000","last_became_active":"2023-07-14T18:10:17.215690+0000","last_became_peered":"2023-07-14T18:10:17.215690+0000","last_unstale":"2023-07-14T18:10:24.089769+0000","last_undegraded":"2023-07-14T18:10:24.089769+0000","last_fullsized":"2023-07-14T18:10:24.089769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:54.597911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.734912+0000","last_change":"2023-07-14T18:10:18.114043+0000","last_active":"2023-07-14T18:10:23.734912+0000","last_peered":"2023-07-14T18:10:23.734912+0000","last_clean":"2023-07-14T18:10:23.734912+0000","last_became_active":"2023-07-14T18:10:18.113745+0000","last_became_peered":"2023-07-14T18:10:18.113745+0000","last_unstale":"2023-07-14T18:10:23.734912+0000","last_undegraded":"2023-07-14T18:10:23.734912+0000","last_fullsized":"2023-07-14T18:10:23.734912+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:42:18.691206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846054+0000","last_change":"2023-07-14T18:10:22.501904+0000","last_active":"2023-07-14T18:10:23.846054+0000","last_peered":"2023-07-14T18:10:23.846054+0000","last_clean":"2023-07-14T18:10:23.846054+0000","last_became_active":"2023-07-14T18:10:22.501500+0000","last_became_peered":"2023-07-14T18:10:22.501500+0000","last_unstale":"2023-07-14T18:10:23.846054+0000","last_undegraded":"2023-07-14T18:10:23.846054+0000","last_fullsized":"2023-07-14T18:10:23.846054+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:27:26.164775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090043+0000","last_change":"2023-07-14T18:10:18.114655+0000","last_active":"2023-07-14T18:10:24.090043+0000","last_peered":"2023-07-14T18:10:24.090043+0000","last_clean":"2023-07-14T18:10:24.090043+0000","last_became_active":"2023-07-14T18:10:18.113984+0000","last_became_peered":"2023-07-14T18:10:18.113984+0000","last_unstale":"2023-07-14T18:10:24.090043+0000","last_undegraded":"2023-07-14T18:10:24.090043+0000","last_fullsized":"2023-07-14T18:10:24.090043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:51:36.108289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:11:22.502016+0000","last_change":"2023-07-14T18:10:17.212612+0000","last_active":"2023-07-14T18:11:22.502016+0000","last_peered":"2023-07-14T18:11:22.502016+0000","last_clean":"2023-07-14T18:11:22.502016+0000","last_became_active":"2023-07-14T18:10:17.212282+0000","last_became_peered":"2023-07-14T18:10:17.212282+0000","last_unstale":"2023-07-14T18:11:22.502016+0000","last_undegraded":"2023-07-14T18:11:22.502016+0000","last_fullsized":"2023-07-14T18:11:22.502016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:12:46.994247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089136+0000","last_change":"2023-07-14T18:10:22.449831+0000","last_active":"2023-07-14T18:10:24.089136+0000","last_peered":"2023-07-14T18:10:24.089136+0000","last_clean":"2023-07-14T18:10:24.089136+0000","last_became_active":"2023-07-14T18:10:22.449380+0000","last_became_peered":"2023-07-14T18:10:22.449380+0000","last_unstale":"2023-07-14T18:10:24.089136+0000","last_undegraded":"2023-07-14T18:10:24.089136+0000","last_fullsized":"2023-07-14T18:10:24.089136+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:33.180987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088819+0000","last_change":"2023-07-14T18:10:18.107041+0000","last_active":"2023-07-14T18:10:24.088819+0000","last_peered":"2023-07-14T18:10:24.088819+0000","last_clean":"2023-07-14T18:10:24.088819+0000","last_became_active":"2023-07-14T18:10:18.106715+0000","last_became_peered":"2023-07-14T18:10:18.106715+0000","last_unstale":"2023-07-14T18:10:24.088819+0000","last_undegraded":"2023-07-14T18:10:24.088819+0000","last_fullsized":"2023-07-14T18:10:24.088819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:49.905307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428953+0000","last_change":"2023-07-14T18:10:17.205525+0000","last_active":"2023-07-14T18:10:37.428953+0000","last_peered":"2023-07-14T18:10:37.428953+0000","last_clean":"2023-07-14T18:10:37.428953+0000","last_became_active":"2023-07-14T18:10:17.205164+0000","last_became_peered":"2023-07-14T18:10:17.205164+0000","last_unstale":"2023-07-14T18:10:37.428953+0000","last_undegraded":"2023-07-14T18:10:37.428953+0000","last_fullsized":"2023-07-14T18:10:37.428953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:06:06.720104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088564+0000","last_change":"2023-07-14T18:09:18.495776+0000","last_active":"2023-07-14T18:10:24.088564+0000","last_peered":"2023-07-14T18:10:24.088564+0000","last_clean":"2023-07-14T18:10:24.088564+0000","last_became_active":"2023-07-14T18:09:18.495326+0000","last_became_peered":"2023-07-14T18:09:18.495326+0000","last_unstale":"2023-07-14T18:10:24.088564+0000","last_undegraded":"2023-07-14T18:10:24.088564+0000","last_fullsized":"2023-07-14T18:10:24.088564+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:06:31.483975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:06:31.483975+0000","last_clean_scrub_stamp":"2023-07-14T18:06:31.483975+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:42:18.539340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090230+0000","last_change":"2023-07-14T18:10:22.501067+0000","last_active":"2023-07-14T18:10:24.090230+0000","last_peered":"2023-07-14T18:10:24.090230+0000","last_clean":"2023-07-14T18:10:24.090230+0000","last_became_active":"2023-07-14T18:10:22.500337+0000","last_became_peered":"2023-07-14T18:10:22.500337+0000","last_unstale":"2023-07-14T18:10:24.090230+0000","last_undegraded":"2023-07-14T18:10:24.090230+0000","last_fullsized":"2023-07-14T18:10:24.090230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:43.900787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845551+0000","last_change":"2023-07-14T18:10:18.107299+0000","last_active":"2023-07-14T18:10:23.845551+0000","last_peered":"2023-07-14T18:10:23.845551+0000","last_clean":"2023-07-14T18:10:23.845551+0000","last_became_active":"2023-07-14T18:10:18.106986+0000","last_became_peered":"2023-07-14T18:10:18.106986+0000","last_unstale":"2023-07-14T18:10:23.845551+0000","last_undegraded":"2023-07-14T18:10:23.845551+0000","last_fullsized":"2023-07-14T18:10:23.845551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:47.211264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.429208+0000","last_change":"2023-07-14T18:10:17.217895+0000","last_active":"2023-07-14T18:10:37.429208+0000","last_peered":"2023-07-14T18:10:37.429208+0000","last_clean":"2023-07-14T18:10:37.429208+0000","last_became_active":"2023-07-14T18:10:17.217548+0000","last_became_peered":"2023-07-14T18:10:17.217548+0000","last_unstale":"2023-07-14T18:10:37.429208+0000","last_undegraded":"2023-07-14T18:10:37.429208+0000","last_fullsized":"2023-07-14T18:10:37.429208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:29:08.392282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088848+0000","last_change":"2023-07-14T18:10:22.453140+0000","last_active":"2023-07-14T18:10:24.088848+0000","last_peered":"2023-07-14T18:10:24.088848+0000","last_clean":"2023-07-14T18:10:24.088848+0000","last_became_active":"2023-07-14T18:10:22.452731+0000","last_became_peered":"2023-07-14T18:10:22.452731+0000","last_unstale":"2023-07-14T18:10:24.088848+0000","last_undegraded":"2023-07-14T18:10:24.088848+0000","last_fullsized":"2023-07-14T18:10:24.088848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:40.973261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980328+0000","last_change":"2023-07-14T18:10:17.213431+0000","last_active":"2023-07-14T18:10:36.980328+0000","last_peered":"2023-07-14T18:10:36.980328+0000","last_clean":"2023-07-14T18:10:36.980328+0000","last_became_active":"2023-07-14T18:10:17.213125+0000","last_became_peered":"2023-07-14T18:10:17.213125+0000","last_unstale":"2023-07-14T18:10:36.980328+0000","last_undegraded":"2023-07-14T18:10:36.980328+0000","last_fullsized":"2023-07-14T18:10:36.980328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:53:36.345583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846032+0000","last_change":"2023-07-14T18:10:18.110593+0000","last_active":"2023-07-14T18:10:23.846032+0000","last_peered":"2023-07-14T18:10:23.846032+0000","last_clean":"2023-07-14T18:10:23.846032+0000","last_became_active":"2023-07-14T18:10:18.110220+0000","last_became_peered":"2023-07-14T18:10:18.110220+0000","last_unstale":"2023-07-14T18:10:23.846032+0000","last_undegraded":"2023-07-14T18:10:23.846032+0000","last_fullsized":"2023-07-14T18:10:23.846032+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:58:18.893324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088540+0000","last_change":"2023-07-14T18:10:22.500809+0000","last_active":"2023-07-14T18:10:24.088540+0000","last_peered":"2023-07-14T18:10:24.088540+0000","last_clean":"2023-07-14T18:10:24.088540+0000","last_became_active":"2023-07-14T18:10:22.500387+0000","last_became_peered":"2023-07-14T18:10:22.500387+0000","last_unstale":"2023-07-14T18:10:24.088540+0000","last_undegraded":"2023-07-14T18:10:24.088540+0000","last_fullsized":"2023-07-14T18:10:24.088540+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:26.241462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'5","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:41.563706+0000","last_change":"2023-07-14T18:10:17.209632+0000","last_active":"2023-07-14T18:10:41.563706+0000","last_peered":"2023-07-14T18:10:41.563706+0000","last_clean":"2023-07-14T18:10:41.563706+0000","last_became_active":"2023-07-14T18:10:17.209274+0000","last_became_peered":"2023-07-14T18:10:17.209274+0000","last_unstale":"2023-07-14T18:10:41.563706+0000","last_undegraded":"2023-07-14T18:10:41.563706+0000","last_fullsized":"2023-07-14T18:10:41.563706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:49:02.203177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089197+0000","last_change":"2023-07-14T18:10:18.119828+0000","last_active":"2023-07-14T18:10:24.089197+0000","last_peered":"2023-07-14T18:10:24.089197+0000","last_clean":"2023-07-14T18:10:24.089197+0000","last_became_active":"2023-07-14T18:10:18.119551+0000","last_became_peered":"2023-07-14T18:10:18.119551+0000","last_unstale":"2023-07-14T18:10:24.089197+0000","last_undegraded":"2023-07-14T18:10:24.089197+0000","last_fullsized":"2023-07-14T18:10:24.089197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:59:11.985338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549754+0000","last_change":"2023-07-14T18:10:22.450325+0000","last_active":"2023-07-14T18:10:23.549754+0000","last_peered":"2023-07-14T18:10:23.549754+0000","last_clean":"2023-07-14T18:10:23.549754+0000","last_became_active":"2023-07-14T18:10:22.449492+0000","last_became_peered":"2023-07-14T18:10:22.449492+0000","last_unstale":"2023-07-14T18:10:23.549754+0000","last_undegraded":"2023-07-14T18:10:23.549754+0000","last_fullsized":"2023-07-14T18:10:23.549754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:37:37.829472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981938+0000","last_change":"2023-07-14T18:10:17.212891+0000","last_active":"2023-07-14T18:10:36.981938+0000","last_peered":"2023-07-14T18:10:36.981938+0000","last_clean":"2023-07-14T18:10:36.981938+0000","last_became_active":"2023-07-14T18:10:17.212573+0000","last_became_peered":"2023-07-14T18:10:17.212573+0000","last_unstale":"2023-07-14T18:10:36.981938+0000","last_undegraded":"2023-07-14T18:10:36.981938+0000","last_fullsized":"2023-07-14T18:10:36.981938+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:38:07.247348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440729+0000","last_change":"2023-07-14T18:10:18.113389+0000","last_active":"2023-07-14T18:10:23.440729+0000","last_peered":"2023-07-14T18:10:23.440729+0000","last_clean":"2023-07-14T18:10:23.440729+0000","last_became_active":"2023-07-14T18:10:18.113053+0000","last_became_peered":"2023-07-14T18:10:18.113053+0000","last_unstale":"2023-07-14T18:10:23.440729+0000","last_undegraded":"2023-07-14T18:10:23.440729+0000","last_fullsized":"2023-07-14T18:10:23.440729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:29:31.077735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438597+0000","last_change":"2023-07-14T18:10:22.500749+0000","last_active":"2023-07-14T18:10:23.438597+0000","last_peered":"2023-07-14T18:10:23.438597+0000","last_clean":"2023-07-14T18:10:23.438597+0000","last_became_active":"2023-07-14T18:10:22.500357+0000","last_became_peered":"2023-07-14T18:10:22.500357+0000","last_unstale":"2023-07-14T18:10:23.438597+0000","last_undegraded":"2023-07-14T18:10:23.438597+0000","last_fullsized":"2023-07-14T18:10:23.438597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:05:10.947498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980089+0000","last_change":"2023-07-14T18:10:17.212501+0000","last_active":"2023-07-14T18:10:36.980089+0000","last_peered":"2023-07-14T18:10:36.980089+0000","last_clean":"2023-07-14T18:10:36.980089+0000","last_became_active":"2023-07-14T18:10:17.212171+0000","last_became_peered":"2023-07-14T18:10:17.212171+0000","last_unstale":"2023-07-14T18:10:36.980089+0000","last_undegraded":"2023-07-14T18:10:36.980089+0000","last_fullsized":"2023-07-14T18:10:36.980089+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:37.208644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438420+0000","last_change":"2023-07-14T18:10:18.111692+0000","last_active":"2023-07-14T18:10:23.438420+0000","last_peered":"2023-07-14T18:10:23.438420+0000","last_clean":"2023-07-14T18:10:23.438420+0000","last_became_active":"2023-07-14T18:10:18.111424+0000","last_became_peered":"2023-07-14T18:10:18.111424+0000","last_unstale":"2023-07-14T18:10:23.438420+0000","last_undegraded":"2023-07-14T18:10:23.438420+0000","last_fullsized":"2023-07-14T18:10:23.438420+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:06.610911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089989+0000","last_change":"2023-07-14T18:10:22.456773+0000","last_active":"2023-07-14T18:10:24.089989+0000","last_peered":"2023-07-14T18:10:24.089989+0000","last_clean":"2023-07-14T18:10:24.089989+0000","last_became_active":"2023-07-14T18:10:22.456309+0000","last_became_peered":"2023-07-14T18:10:22.456309+0000","last_unstale":"2023-07-14T18:10:24.089989+0000","last_undegraded":"2023-07-14T18:10:24.089989+0000","last_fullsized":"2023-07-14T18:10:24.089989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:29:09.644386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.980843+0000","last_change":"2023-07-14T18:10:17.205109+0000","last_active":"2023-07-14T18:10:36.980843+0000","last_peered":"2023-07-14T18:10:36.980843+0000","last_clean":"2023-07-14T18:10:36.980843+0000","last_became_active":"2023-07-14T18:10:17.204644+0000","last_became_peered":"2023-07-14T18:10:17.204644+0000","last_unstale":"2023-07-14T18:10:36.980843+0000","last_undegraded":"2023-07-14T18:10:36.980843+0000","last_fullsized":"2023-07-14T18:10:36.980843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:08.967260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088692+0000","last_change":"2023-07-14T18:10:18.109143+0000","last_active":"2023-07-14T18:10:24.088692+0000","last_peered":"2023-07-14T18:10:24.088692+0000","last_clean":"2023-07-14T18:10:24.088692+0000","last_became_active":"2023-07-14T18:10:18.108841+0000","last_became_peered":"2023-07-14T18:10:18.108841+0000","last_unstale":"2023-07-14T18:10:24.088692+0000","last_undegraded":"2023-07-14T18:10:24.088692+0000","last_fullsized":"2023-07-14T18:10:24.088692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:30:16.212954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438346+0000","last_change":"2023-07-14T18:10:22.450564+0000","last_active":"2023-07-14T18:10:23.438346+0000","last_peered":"2023-07-14T18:10:23.438346+0000","last_clean":"2023-07-14T18:10:23.438346+0000","last_became_active":"2023-07-14T18:10:22.449801+0000","last_became_peered":"2023-07-14T18:10:22.449801+0000","last_unstale":"2023-07-14T18:10:23.438346+0000","last_undegraded":"2023-07-14T18:10:23.438346+0000","last_fullsized":"2023-07-14T18:10:23.438346+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:00:04.711449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549788+0000","last_change":"2023-07-14T18:10:17.211522+0000","last_active":"2023-07-14T18:10:23.549788+0000","last_peered":"2023-07-14T18:10:23.549788+0000","last_clean":"2023-07-14T18:10:23.549788+0000","last_became_active":"2023-07-14T18:10:17.211163+0000","last_became_peered":"2023-07-14T18:10:17.211163+0000","last_unstale":"2023-07-14T18:10:23.549788+0000","last_undegraded":"2023-07-14T18:10:23.549788+0000","last_fullsized":"2023-07-14T18:10:23.549788+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:14:47.230105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089155+0000","last_change":"2023-07-14T18:10:18.111885+0000","last_active":"2023-07-14T18:10:24.089155+0000","last_peered":"2023-07-14T18:10:24.089155+0000","last_clean":"2023-07-14T18:10:24.089155+0000","last_became_active":"2023-07-14T18:10:18.111520+0000","last_became_peered":"2023-07-14T18:10:18.111520+0000","last_unstale":"2023-07-14T18:10:24.089155+0000","last_undegraded":"2023-07-14T18:10:24.089155+0000","last_fullsized":"2023-07-14T18:10:24.089155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:06:29.717829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.438617+0000","last_change":"2023-07-14T18:10:22.500612+0000","last_active":"2023-07-14T18:10:23.438617+0000","last_peered":"2023-07-14T18:10:23.438617+0000","last_clean":"2023-07-14T18:10:23.438617+0000","last_became_active":"2023-07-14T18:10:22.500219+0000","last_became_peered":"2023-07-14T18:10:22.500219+0000","last_unstale":"2023-07-14T18:10:23.438617+0000","last_undegraded":"2023-07-14T18:10:23.438617+0000","last_fullsized":"2023-07-14T18:10:23.438617+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:09.251191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:36.981056+0000","last_change":"2023-07-14T18:10:17.214307+0000","last_active":"2023-07-14T18:10:36.981056+0000","last_peered":"2023-07-14T18:10:36.981056+0000","last_clean":"2023-07-14T18:10:36.981056+0000","last_became_active":"2023-07-14T18:10:17.214004+0000","last_became_peered":"2023-07-14T18:10:17.214004+0000","last_unstale":"2023-07-14T18:10:36.981056+0000","last_undegraded":"2023-07-14T18:10:36.981056+0000","last_fullsized":"2023-07-14T18:10:36.981056+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:50:59.158356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089823+0000","last_change":"2023-07-14T18:10:18.120504+0000","last_active":"2023-07-14T18:10:24.089823+0000","last_peered":"2023-07-14T18:10:24.089823+0000","last_clean":"2023-07-14T18:10:24.089823+0000","last_became_active":"2023-07-14T18:10:18.120133+0000","last_became_peered":"2023-07-14T18:10:18.120133+0000","last_unstale":"2023-07-14T18:10:24.089823+0000","last_undegraded":"2023-07-14T18:10:24.089823+0000","last_fullsized":"2023-07-14T18:10:24.089823+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:54.324914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089420+0000","last_change":"2023-07-14T18:10:22.453040+0000","last_active":"2023-07-14T18:10:24.089420+0000","last_peered":"2023-07-14T18:10:24.089420+0000","last_clean":"2023-07-14T18:10:24.089420+0000","last_became_active":"2023-07-14T18:10:22.452694+0000","last_became_peered":"2023-07-14T18:10:22.452694+0000","last_unstale":"2023-07-14T18:10:24.089420+0000","last_undegraded":"2023-07-14T18:10:24.089420+0000","last_fullsized":"2023-07-14T18:10:24.089420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:43:06.299843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:37.428686+0000","last_change":"2023-07-14T18:10:17.212480+0000","last_active":"2023-07-14T18:10:37.428686+0000","last_peered":"2023-07-14T18:10:37.428686+0000","last_clean":"2023-07-14T18:10:37.428686+0000","last_became_active":"2023-07-14T18:10:17.212080+0000","last_became_peered":"2023-07-14T18:10:17.212080+0000","last_unstale":"2023-07-14T18:10:37.428686+0000","last_undegraded":"2023-07-14T18:10:37.428686+0000","last_fullsized":"2023-07-14T18:10:37.428686+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:19:44.529873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089506+0000","last_change":"2023-07-14T18:10:18.121738+0000","last_active":"2023-07-14T18:10:24.089506+0000","last_peered":"2023-07-14T18:10:24.089506+0000","last_clean":"2023-07-14T18:10:24.089506+0000","last_became_active":"2023-07-14T18:10:18.121186+0000","last_became_peered":"2023-07-14T18:10:18.121186+0000","last_unstale":"2023-07-14T18:10:24.089506+0000","last_undegraded":"2023-07-14T18:10:24.089506+0000","last_fullsized":"2023-07-14T18:10:24.089506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:40:16.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440225+0000","last_change":"2023-07-14T18:10:22.461051+0000","last_active":"2023-07-14T18:10:23.440225+0000","last_peered":"2023-07-14T18:10:23.440225+0000","last_clean":"2023-07-14T18:10:23.440225+0000","last_became_active":"2023-07-14T18:10:22.460663+0000","last_became_peered":"2023-07-14T18:10:22.460663+0000","last_unstale":"2023-07-14T18:10:23.440225+0000","last_undegraded":"2023-07-14T18:10:23.440225+0000","last_fullsized":"2023-07-14T18:10:23.440225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:58:08.355441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.090658+0000","last_change":"2023-07-14T18:10:17.211686+0000","last_active":"2023-07-14T18:10:24.090658+0000","last_peered":"2023-07-14T18:10:24.090658+0000","last_clean":"2023-07-14T18:10:24.090658+0000","last_became_active":"2023-07-14T18:10:17.211367+0000","last_became_peered":"2023-07-14T18:10:17.211367+0000","last_unstale":"2023-07-14T18:10:24.090658+0000","last_undegraded":"2023-07-14T18:10:24.090658+0000","last_fullsized":"2023-07-14T18:10:24.090658+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:50.829623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089608+0000","last_change":"2023-07-14T18:10:18.116960+0000","last_active":"2023-07-14T18:10:24.089608+0000","last_peered":"2023-07-14T18:10:24.089608+0000","last_clean":"2023-07-14T18:10:24.089608+0000","last_became_active":"2023-07-14T18:10:18.116416+0000","last_became_peered":"2023-07-14T18:10:18.116416+0000","last_unstale":"2023-07-14T18:10:24.089608+0000","last_undegraded":"2023-07-14T18:10:24.089608+0000","last_fullsized":"2023-07-14T18:10:24.089608+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:33.066056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089552+0000","last_change":"2023-07-14T18:10:22.450045+0000","last_active":"2023-07-14T18:10:24.089552+0000","last_peered":"2023-07-14T18:10:24.089552+0000","last_clean":"2023-07-14T18:10:24.089552+0000","last_became_active":"2023-07-14T18:10:22.449621+0000","last_became_peered":"2023-07-14T18:10:22.449621+0000","last_unstale":"2023-07-14T18:10:24.089552+0000","last_undegraded":"2023-07-14T18:10:24.089552+0000","last_fullsized":"2023-07-14T18:10:24.089552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:21.421656+0000","last_clean_scrub_stamp":"2023-07-14T18:10:21.421656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:48:23.837464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846188+0000","last_change":"2023-07-14T18:10:17.210691+0000","last_active":"2023-07-14T18:10:23.846188+0000","last_peered":"2023-07-14T18:10:23.846188+0000","last_clean":"2023-07-14T18:10:23.846188+0000","last_became_active":"2023-07-14T18:10:17.210391+0000","last_became_peered":"2023-07-14T18:10:17.210391+0000","last_unstale":"2023-07-14T18:10:23.846188+0000","last_undegraded":"2023-07-14T18:10:23.846188+0000","last_fullsized":"2023-07-14T18:10:23.846188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:16.183034+0000","last_clean_scrub_stamp":"2023-07-14T18:10:16.183034+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:20:04.361243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089271+0000","last_change":"2023-07-14T18:10:18.115347+0000","last_active":"2023-07-14T18:10:24.089271+0000","last_peered":"2023-07-14T18:10:24.089271+0000","last_clean":"2023-07-14T18:10:24.089271+0000","last_became_active":"2023-07-14T18:10:18.115027+0000","last_became_peered":"2023-07-14T18:10:18.115027+0000","last_unstale":"2023-07-14T18:10:24.089271+0000","last_undegraded":"2023-07-14T18:10:24.089271+0000","last_fullsized":"2023-07-14T18:10:24.089271+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:33:02.993257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088765+0000","last_change":"2023-07-14T18:10:18.103185+0000","last_active":"2023-07-14T18:10:24.088765+0000","last_peered":"2023-07-14T18:10:24.088765+0000","last_clean":"2023-07-14T18:10:24.088765+0000","last_became_active":"2023-07-14T18:10:18.102800+0000","last_became_peered":"2023-07-14T18:10:18.102800+0000","last_unstale":"2023-07-14T18:10:24.088765+0000","last_undegraded":"2023-07-14T18:10:24.088765+0000","last_fullsized":"2023-07-14T18:10:24.088765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:35:22.321627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550057+0000","last_change":"2023-07-14T18:10:18.113501+0000","last_active":"2023-07-14T18:10:23.550057+0000","last_peered":"2023-07-14T18:10:23.550057+0000","last_clean":"2023-07-14T18:10:23.550057+0000","last_became_active":"2023-07-14T18:10:18.113205+0000","last_became_peered":"2023-07-14T18:10:18.113205+0000","last_unstale":"2023-07-14T18:10:23.550057+0000","last_undegraded":"2023-07-14T18:10:23.550057+0000","last_fullsized":"2023-07-14T18:10:23.550057+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:13:56.420803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.549722+0000","last_change":"2023-07-14T18:10:18.114282+0000","last_active":"2023-07-14T18:10:23.549722+0000","last_peered":"2023-07-14T18:10:23.549722+0000","last_clean":"2023-07-14T18:10:23.549722+0000","last_became_active":"2023-07-14T18:10:18.113968+0000","last_became_peered":"2023-07-14T18:10:18.113968+0000","last_unstale":"2023-07-14T18:10:23.549722+0000","last_undegraded":"2023-07-14T18:10:23.549722+0000","last_fullsized":"2023-07-14T18:10:23.549722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:36.356073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.440436+0000","last_change":"2023-07-14T18:10:18.116781+0000","last_active":"2023-07-14T18:10:23.440436+0000","last_peered":"2023-07-14T18:10:23.440436+0000","last_clean":"2023-07-14T18:10:23.440436+0000","last_became_active":"2023-07-14T18:10:18.116352+0000","last_became_peered":"2023-07-14T18:10:18.116352+0000","last_unstale":"2023-07-14T18:10:23.440436+0000","last_undegraded":"2023-07-14T18:10:23.440436+0000","last_fullsized":"2023-07-14T18:10:23.440436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:48:03.670475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.846088+0000","last_change":"2023-07-14T18:10:18.112291+0000","last_active":"2023-07-14T18:10:23.846088+0000","last_peered":"2023-07-14T18:10:23.846088+0000","last_clean":"2023-07-14T18:10:23.846088+0000","last_became_active":"2023-07-14T18:10:18.111989+0000","last_became_peered":"2023-07-14T18:10:18.111989+0000","last_unstale":"2023-07-14T18:10:23.846088+0000","last_undegraded":"2023-07-14T18:10:23.846088+0000","last_fullsized":"2023-07-14T18:10:23.846088+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:11:33.924006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089268+0000","last_change":"2023-07-14T18:10:18.112316+0000","last_active":"2023-07-14T18:10:24.089268+0000","last_peered":"2023-07-14T18:10:24.089268+0000","last_clean":"2023-07-14T18:10:24.089268+0000","last_became_active":"2023-07-14T18:10:18.112030+0000","last_became_peered":"2023-07-14T18:10:18.112030+0000","last_unstale":"2023-07-14T18:10:24.089268+0000","last_undegraded":"2023-07-14T18:10:24.089268+0000","last_fullsized":"2023-07-14T18:10:24.089268+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:14:55.405097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.550228+0000","last_change":"2023-07-14T18:10:18.116343+0000","last_active":"2023-07-14T18:10:23.550228+0000","last_peered":"2023-07-14T18:10:23.550228+0000","last_clean":"2023-07-14T18:10:23.550228+0000","last_became_active":"2023-07-14T18:10:18.116041+0000","last_became_peered":"2023-07-14T18:10:18.116041+0000","last_unstale":"2023-07-14T18:10:23.550228+0000","last_undegraded":"2023-07-14T18:10:23.550228+0000","last_fullsized":"2023-07-14T18:10:23.550228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:43:43.627634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089357+0000","last_change":"2023-07-14T18:10:18.117580+0000","last_active":"2023-07-14T18:10:24.089357+0000","last_peered":"2023-07-14T18:10:24.089357+0000","last_clean":"2023-07-14T18:10:24.089357+0000","last_became_active":"2023-07-14T18:10:18.117249+0000","last_became_peered":"2023-07-14T18:10:18.117249+0000","last_unstale":"2023-07-14T18:10:24.089357+0000","last_undegraded":"2023-07-14T18:10:24.089357+0000","last_fullsized":"2023-07-14T18:10:24.089357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:42.562595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089822+0000","last_change":"2023-07-14T18:10:18.107276+0000","last_active":"2023-07-14T18:10:24.089822+0000","last_peered":"2023-07-14T18:10:24.089822+0000","last_clean":"2023-07-14T18:10:24.089822+0000","last_became_active":"2023-07-14T18:10:18.107005+0000","last_became_peered":"2023-07-14T18:10:18.107005+0000","last_unstale":"2023-07-14T18:10:24.089822+0000","last_undegraded":"2023-07-14T18:10:24.089822+0000","last_fullsized":"2023-07-14T18:10:24.089822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:13.352444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089145+0000","last_change":"2023-07-14T18:10:18.107921+0000","last_active":"2023-07-14T18:10:24.089145+0000","last_peered":"2023-07-14T18:10:24.089145+0000","last_clean":"2023-07-14T18:10:24.089145+0000","last_became_active":"2023-07-14T18:10:18.107637+0000","last_became_peered":"2023-07-14T18:10:18.107637+0000","last_unstale":"2023-07-14T18:10:24.089145+0000","last_undegraded":"2023-07-14T18:10:24.089145+0000","last_fullsized":"2023-07-14T18:10:24.089145+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:32.687817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.089709+0000","last_change":"2023-07-14T18:10:18.120125+0000","last_active":"2023-07-14T18:10:24.089709+0000","last_peered":"2023-07-14T18:10:24.089709+0000","last_clean":"2023-07-14T18:10:24.089709+0000","last_became_active":"2023-07-14T18:10:18.119814+0000","last_became_peered":"2023-07-14T18:10:18.119814+0000","last_unstale":"2023-07-14T18:10:24.089709+0000","last_undegraded":"2023-07-14T18:10:24.089709+0000","last_fullsized":"2023-07-14T18:10:24.089709+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:37:22.531320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:23.845885+0000","last_change":"2023-07-14T18:10:18.113155+0000","last_active":"2023-07-14T18:10:23.845885+0000","last_peered":"2023-07-14T18:10:23.845885+0000","last_clean":"2023-07-14T18:10:23.845885+0000","last_became_active":"2023-07-14T18:10:18.112837+0000","last_became_peered":"2023-07-14T18:10:18.112837+0000","last_unstale":"2023-07-14T18:10:23.845885+0000","last_undegraded":"2023-07-14T18:10:23.845885+0000","last_fullsized":"2023-07-14T18:10:23.845885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:56:13.124803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:10:24.088936+0000","last_change":"2023-07-14T18:10:18.112290+0000","last_active":"2023-07-14T18:10:24.088936+0000","last_peered":"2023-07-14T18:10:24.088936+0000","last_clean":"2023-07-14T18:10:24.088936+0000","last_became_active":"2023-07-14T18:10:18.111988+0000","last_became_peered":"2023-07-14T18:10:18.111988+0000","last_unstale":"2023-07-14T18:10:24.088936+0000","last_undegraded":"2023-07-14T18:10:24.088936+0000","last_fullsized":"2023-07-14T18:10:24.088936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:10:17.189998+0000","last_clean_scrub_stamp":"2023-07-14T18:10:17.189998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:26.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62845,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":295,"num_read_kb":214,"num_write":154,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":188535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449112,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":470478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.36399999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.314},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":4,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.66400000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.435}]},{"osd":6,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.55500000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.52500000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.496}]},{"osd":10,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.76100000000000001}]}]},{"osd":10,"up_from":71,"seq":304942678044,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1116308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.57399999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.48299999999999998,"15min":0.48699999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77000000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82099999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.64500000000000002,"15min":0.64400000000000002},"min":{"1min":0.435,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86199999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52900000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.85099999999999998,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.42299999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.622,"15min":0.63400000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.35299999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89500000000000002,"5min":1.042,"15min":1.042},"last":0.35299999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.71599999999999997,"15min":0.72099999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96199999999999997,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.73199999999999998,"15min":0.73499999999999999},"min":{"1min":0.44900000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.44900000000000001}]},{"osd":6,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.73699999999999999,"15min":0.748},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.94399999999999995,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.76700000000000002,"15min":0.77300000000000002},"min":{"1min":0.55200000000000005,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.036,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.69299999999999995}]},{"osd":8,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.79800000000000004,"15min":0.80200000000000005},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.81100000000000005,"15min":0.82099999999999995},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.0229999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.53800000000000003}]},{"osd":9,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.80700000000000005,"15min":0.81100000000000005},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1399999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.81599999999999995,"15min":0.82999999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.069,"5min":1.329,"15min":1.329},"last":0.623}]},{"osd":11,"last update":"Fri Jul 14 18:10:41 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.66200000000000003}]}]},{"osd":9,"up_from":66,"seq":283467841569,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":474810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.442,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.53600000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.44700000000000001,"15min":0.45800000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.58399999999999996,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.28699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.38200000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83699999999999997,"5min":0.875,"15min":0.875},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82899999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.42499999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.59599999999999997,"15min":0.60299999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81999999999999995,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.57199999999999995}]},{"osd":3,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.496,"15min":0.503},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68400000000000005,"5min":0.75,"15min":0.75},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.51000000000000001,"15min":0.52200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66800000000000004,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.313}]},{"osd":4,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.63500000000000001,"15min":0.64700000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76800000000000002,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.373},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67500000000000004,"15min":0.68899999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.434}]},{"osd":5,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64900000000000002,"15min":0.65800000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.91000000000000003,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.68200000000000005,"15min":0.69799999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.80200000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58199999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:09:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.71899999999999997,"15min":0.73499999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.72299999999999998,"15min":0.73899999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.871,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.44800000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.72999999999999998,"15min":0.748},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.83199999999999996,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73799999999999999,"15min":0.749},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.97999999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.64900000000000002}]},{"osd":10,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.74399999999999999,"15min":0.76100000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.871,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.73699999999999999,"15min":0.75600000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.52300000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":8,"up_from":58,"seq":249108103205,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":470303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.41499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56200000000000006,"15min":0.56499999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.376}]},{"osd":1,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61899999999999999,"15min":0.61599999999999999},"min":{"1min":0.45900000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.51000000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.42999999999999999,"15min":0.42299999999999999},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.312},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46999999999999997,"15min":0.46899999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.35699999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.433,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79700000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.40000000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60399999999999998,"15min":0.59899999999999998},"min":{"1min":0.39900000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64100000000000001,"15min":0.63900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.36299999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.45900000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.52300000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.441},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72099999999999997,"15min":0.72599999999999998},"min":{"1min":0.44600000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.53100000000000003,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.626},{"interface":"front","average":{"1min":0.746,"5min":0.70099999999999996,"15min":0.69299999999999995},"min":{"1min":0.501,"5min":0.439,"15min":0.439},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.61799999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.70299999999999996,"15min":0.69099999999999995},"min":{"1min":0.47299999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.046,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.73599999999999999,"15min":0.72799999999999998},"min":{"1min":0.55500000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0649999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53300000000000003}]},{"osd":10,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.753,"15min":0.72499999999999998},"min":{"1min":0.622,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.754,"15min":0.73099999999999998},"min":{"1min":0.74199999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0269999999999999,"5min":1.135,"15min":1.135},"last":0.60799999999999998}]},{"osd":11,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.55600000000000005}]}]},{"osd":7,"up_from":50,"seq":214748364841,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1066836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.41399999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.42799999999999999,"15min":0.41899999999999998},"min":{"1min":0.314,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.373},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.45100000000000001,"15min":0.44},"min":{"1min":0.29299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.59799999999999998,"15min":0.58099999999999996},"min":{"1min":0.41199999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60299999999999998,"15min":0.58699999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.441}]},{"osd":3,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.55900000000000005,"15min":0.54300000000000004},"min":{"1min":0.41099999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.75800000000000001,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.59099999999999997,"15min":0.58099999999999996},"min":{"1min":0.43099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.77400000000000002,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34300000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77800000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52400000000000002,"15min":0.51400000000000001},"min":{"1min":0.35499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.5}]},{"osd":5,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.65600000000000003,"15min":0.63200000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.68600000000000005,"15min":0.68000000000000005},"min":{"1min":0.47099999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.42799999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.86299999999999999,"5min":1.024,"15min":1.024},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.67300000000000004,"15min":0.65100000000000002},"min":{"1min":0.52000000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91000000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52000000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:10:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.66700000000000004,"15min":0.67300000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.877,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.39900000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72799999999999998,"15min":0.73199999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.51100000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72799999999999998,"15min":0.72499999999999998},"min":{"1min":0.49199999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96599999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77400000000000002,"15min":0.76100000000000001},"min":{"1min":0.52000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.58199999999999996}]},{"osd":11,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.56499999999999995}]}]},{"osd":6,"up_from":45,"seq":193273528365,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":465890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.38600000000000001,"15min":0.38200000000000001},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.307},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.27700000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.627,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.32700000000000001}]},{"osd":1,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.442,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58499999999999996,"15min":0.58099999999999996},"min":{"1min":0.438,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82299999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.622,"15min":0.62},"min":{"1min":0.45900000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78800000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.42599999999999999}]},{"osd":3,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.376},{"interface":"front","average":{"1min":0.439,"5min":0.45300000000000001,"15min":0.46000000000000002},"min":{"1min":0.30099999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.44900000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83699999999999997,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.433},{"interface":"front","average":{"1min":0.622,"5min":0.61699999999999999,"15min":0.60999999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83899999999999997,"5min":0.879,"15min":0.879},"last":0.54600000000000004}]},{"osd":5,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.625,"15min":0.629},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86299999999999999,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64000000000000001,"15min":0.63700000000000001},"min":{"1min":0.46999999999999997,"5min":0.433,"15min":0.433},"max":{"1min":0.88,"5min":1.012,"15min":1.012},"last":0.53200000000000003}]},{"osd":7,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.68899999999999995,"15min":0.70099999999999996},"min":{"1min":0.438,"5min":0.436,"15min":0.436},"max":{"1min":0.877,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69999999999999996},"min":{"1min":0.52000000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.86899999999999999,"5min":1.077,"15min":1.077},"last":0.60299999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:10:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.75,"15min":0.76200000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92200000000000004,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.627},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.68200000000000005,"15min":0.69199999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.88500000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59099999999999997}]},{"osd":9,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.39300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.85399999999999998,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61299999999999999,"15min":0.61899999999999999},"min":{"1min":0.40400000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82199999999999995,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51100000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.52900000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.86499999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.54300000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.89600000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.61399999999999999}]},{"osd":11,"last update":"Fri Jul 14 18:11:22 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.66600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":524351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.48499999999999999},"min":{"1min":0.36499999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80400000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.503},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.495},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.82599999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.62}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.622,"15min":0.60399999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.97799999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.60899999999999999},"min":{"1min":0.45300000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.80600000000000005,"5min":1.337,"15min":1.337},"last":0.82799999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:11:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56499999999999995,"15min":0.52300000000000002},"min":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.90700000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54100000000000004,"15min":0.50700000000000001},"min":{"1min":0.41499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.54700000000000004}]},{"osd":4,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.46800000000000003,"15min":0.42999999999999999},"min":{"1min":0.41099999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.48499999999999999,"15min":0.44400000000000001},"min":{"1min":0.42199999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.72599999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.47199999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92900000000000005,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.875},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.64300000000000002,"15min":0.62},"min":{"1min":0.45800000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.86599999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.91900000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:11:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.627,"15min":0.58899999999999997},"min":{"1min":0.49399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96099999999999997,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.64600000000000002,"15min":0.624},"min":{"1min":0.43099999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.76300000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.629,"15min":0.59199999999999997},"min":{"1min":0.51600000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.624,"15min":0.58599999999999997},"min":{"1min":0.47299999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.90700000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70899999999999996,"15min":0.69599999999999995},"min":{"1min":0.59399999999999997,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66100000000000003,"15min":0.65100000000000002},"min":{"1min":0.504,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.79100000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:10:57 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80000000000000004,"15min":0.79500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0189999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.79200000000000004,"15min":0.78400000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.93100000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.72699999999999998,"15min":0.71799999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.88,"5min":0.82399999999999995,"15min":0.81499999999999995},"min":{"1min":0.60699999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":1.036}]},{"osd":11,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":1.0089999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":467278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.55100000000000005,"15min":0.503},"min":{"1min":0.34599999999999997,"5min":0.312,"15min":0.312},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.55200000000000005,"15min":0.51200000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.952,"5min":2.952,"15min":2.952},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52600000000000002,"15min":0.51300000000000001},"min":{"1min":0.41299999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68500000000000005,"5min":0.997,"15min":0.997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53900000000000003,"15min":0.50800000000000001},"min":{"1min":0.40899999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78900000000000003,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.58899999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.32400000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.57399999999999995,"15min":0.53200000000000003},"min":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":2.8889999999999998,"5min":2.8889999999999998,"15min":2.8889999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.53100000000000003},"min":{"1min":0.44,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.495}]},{"osd":5,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70699999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.59099999999999997},"min":{"1min":0.45700000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:10:54 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49299999999999999,"15min":0.47899999999999998},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.59199999999999997,"15min":0.55500000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.6689999999999996,"5min":4.6689999999999996,"15min":4.6689999999999996},"last":0.379}]},{"osd":7,"last update":"Fri Jul 14 18:10:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.65700000000000003,"15min":0.65900000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.92100000000000004,"5min":0.998,"15min":0.998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.621,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85299999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.50800000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64500000000000002,"15min":0.65100000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81299999999999994,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.72099999999999997,"15min":0.72799999999999998},"min":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.88800000000000001,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.68999999999999995}]},{"osd":9,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.66500000000000004,"15min":0.68200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.66200000000000003,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.47599999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:11:20 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.76800000000000002,"15min":0.747},"min":{"1min":0.54500000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":4.4740000000000002,"5min":4.4740000000000002,"15min":4.4740000000000002},"last":0.621},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.60799999999999998}]},{"osd":11,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.67400000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":526043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.67500000000000004,"5min":0.995,"15min":0.995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68899999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54000000000000004,"15min":0.53200000000000003},"min":{"1min":0.39000000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80200000000000005,"5min":1.29,"15min":1.29},"last":0.626},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.34899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.01,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.65800000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80900000000000005,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59099999999999997,"15min":0.55400000000000005},"min":{"1min":0.48299999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91700000000000004,"5min":0.996,"15min":0.996},"last":0.379}]},{"osd":4,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.57899999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.498,"15min":0.47499999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.49199999999999999,"15min":0.46400000000000002},"min":{"1min":0.29699999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.82699999999999996,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Jul 14 18:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.52000000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90000000000000002,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.66200000000000003,"15min":0.64200000000000002},"min":{"1min":0.54000000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.68600000000000005}]},{"osd":7,"last update":"Fri Jul 14 18:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73299999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67400000000000004,"15min":0.66700000000000004},"min":{"1min":0.40699999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.65500000000000003,"15min":0.64700000000000002},"min":{"1min":0.375,"5min":0.371,"15min":0.371},"max":{"1min":0.89100000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.79500000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.73599999999999999,"15min":0.72599999999999998},"min":{"1min":0.61599999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.76400000000000001,"15min":0.76200000000000001},"min":{"1min":0.52000000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.002,"5min":1.524,"15min":1.524},"last":0.69999999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:10:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.748},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.76100000000000001}]},{"osd":11,"last update":"Fri Jul 14 18:11:17 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.82599999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":465753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.36099999999999999,"15min":0.33100000000000002},"min":{"1min":0.25900000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":0.314},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.38400000000000001,"15min":0.35199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.33800000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55300000000000005,"15min":0.53500000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77300000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.36399999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.53700000000000003},"min":{"1min":0.45800000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69099999999999995,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55400000000000005,"15min":0.53500000000000003},"min":{"1min":0.42499999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.57599999999999996}]},{"osd":4,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56799999999999995,"15min":0.52800000000000002},"min":{"1min":0.38700000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.2070000000000001,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57499999999999996,"15min":0.54400000000000004},"min":{"1min":0.40699999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77800000000000002,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.52800000000000002}]},{"osd":5,"last update":"Fri Jul 14 18:10:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.40999999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.626,"15min":0.61199999999999999},"min":{"1min":0.47199999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78300000000000003,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.502},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.9630000000000001,"5min":1.9630000000000001,"15min":1.9630000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.38300000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.69199999999999995,"15min":0.65300000000000002},"min":{"1min":0.41199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":9.0299999999999994,"5min":9.0299999999999994,"15min":9.0299999999999994},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68899999999999995,"15min":0.68500000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.232,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.63}]},{"osd":8,"last update":"Fri Jul 14 18:09:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.64100000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:11:03 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.76300000000000001,"15min":0.78700000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"last":0.48699999999999999}]},{"osd":10,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.72699999999999998,"15min":0.67500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":9.0920000000000005,"5min":9.0920000000000005,"15min":9.0920000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.77700000000000002,"15min":0.79300000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.258,"5min":1.504,"15min":1.504},"last":0.65900000000000003}]},{"osd":11,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.55100000000000005},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":8.9730000000000008,"5min":8.9730000000000008,"15min":8.9730000000000008},"last":0.60399999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953526,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":470882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79400000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51500000000000001,"15min":0.503},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78500000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.36399999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.498,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40400000000000003,"15min":0.38400000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56399999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.377}]},{"osd":2,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59599999999999997,"15min":0.56999999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85399999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.56999999999999995},"min":{"1min":0.42399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56299999999999994,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71799999999999997,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.505},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40000000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81899999999999995,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.53300000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63100000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.439,"5min":0.373,"15min":0.373},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.60199999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:11:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.46100000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88100000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97799999999999998,"5min":0.999,"15min":0.999},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75800000000000001,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59299999999999997,"15min":0.58199999999999996},"min":{"1min":0.47199999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.56100000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:10:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.67600000000000005,"15min":0.67300000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.87,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.71899999999999997,"15min":0.71299999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.70499999999999996,"15min":0.70699999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.71199999999999997,"15min":0.71599999999999997},"min":{"1min":0.57699999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.84399999999999997,"5min":1.024,"15min":1.024},"last":0.624}]},{"osd":10,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.71499999999999997,"15min":0.70899999999999996},"min":{"1min":0.52000000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.71399999999999997,"15min":0.71499999999999997},"min":{"1min":0.56599999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.82199999999999995,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.59199999999999997}]},{"osd":11,"last update":"Fri Jul 14 18:11:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.70399999999999996}]}]},{"osd":5,"up_from":37,"seq":158913790002,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1585152,"data_stored":1062082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86299999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.51500000000000001,"15min":0.52100000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.61099999999999999,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.42699999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57799999999999996,"15min":0.56999999999999995},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.629},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56799999999999995,"15min":0.56100000000000005},"min":{"1min":0.40300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78200000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.42699999999999999,"15min":0.42299999999999999},"min":{"1min":0.30499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.29099999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72399999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.59299999999999997,"15min":0.60099999999999998},"min":{"1min":0.42499999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68700000000000006,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83099999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.60399999999999998},"min":{"1min":0.432,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.90100000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75700000000000001,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.44800000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85799999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.66600000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:11:18 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.628,"15min":0.61799999999999999},"min":{"1min":0.53000000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93300000000000005,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.66200000000000003,"15min":0.65800000000000003},"min":{"1min":0.47699999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.91800000000000004,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.65500000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:10:37 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83899999999999997,"5min":0.873,"15min":0.873},"last":0.64200000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:10:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.66400000000000003,"15min":0.66000000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81999999999999995,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.70199999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:11:25.916 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-14T18:11:25.917 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-14T18:11:25.917 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-14T18:11:25.917 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph health --format=json 2023-07-14T18:11:26.469 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:26.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:26 smithi115 ceph-mon[127451]: pgmap v285: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T18:11:26.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:26 smithi150 ceph-mon[128853]: pgmap v285: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T18:11:26.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:26 smithi187 ceph-mon[129317]: pgmap v285: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T18:11:27.320 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:27 smithi115 ceph-mon[127451]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:11:27.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:27 smithi150 ceph-mon[128853]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:11:27.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:27 smithi187 ceph-mon[129317]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:11:27.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.747+0000 7fe8679e9700 1 -- 172.21.15.115:0/3214898987 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 msgr2=0x7fe8600fe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:27.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.747+0000 7fe8679e9700 1 --2- 172.21.15.115:0/3214898987 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 0x7fe8600fe680 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe854009990 tx=0x7fe854009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:27.791 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.747+0000 7fe8679e9700 1 -- 172.21.15.115:0/3214898987 shutdown_connections 2023-07-14T18:11:27.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.747+0000 7fe8679e9700 1 --2- 172.21.15.115:0/3214898987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8600febc0 0x7fe86010cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:27.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.747+0000 7fe8679e9700 1 --2- 172.21.15.115:0/3214898987 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe860104c70 0x7fe860105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:27.792 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.747+0000 7fe8679e9700 1 --2- 172.21.15.115:0/3214898987 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 0x7fe8600fe680 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:27.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.747+0000 7fe8679e9700 1 -- 172.21.15.115:0/3214898987 >> 172.21.15.115:0/3214898987 conn(0x7fe860075280 msgr2=0x7fe860075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:27.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.747+0000 7fe8679e9700 1 -- 172.21.15.115:0/3214898987 shutdown_connections 2023-07-14T18:11:27.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.747+0000 7fe8679e9700 1 -- 172.21.15.115:0/3214898987 wait complete. 2023-07-14T18:11:27.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe8679e9700 1 Processor -- start 2023-07-14T18:11:27.793 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe8679e9700 1 -- start start 2023-07-14T18:11:27.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe8679e9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 0x7fe8600ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:27.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe8679e9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8600febc0 0x7fe8600ff930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:27.794 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe8679e9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe860104c70 0x7fe8600ffeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:27.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe8679e9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8601a4310 con 0x7fe8600fe210 2023-07-14T18:11:27.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe8679e9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8601a4480 con 0x7fe8600febc0 2023-07-14T18:11:27.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe8679e9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8601a4640 con 0x7fe860104c70 2023-07-14T18:11:27.795 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe865785700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 0x7fe8600ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:27.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe865785700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 0x7fe8600ff3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53558/0 (socket says 172.21.15.115:53558) 2023-07-14T18:11:27.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe865785700 1 -- 172.21.15.115:0/1256197181 learned_addr learned my addr 172.21.15.115:0/1256197181 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:27.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe864f84700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8600febc0 0x7fe8600ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:27.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe865785700 1 -- 172.21.15.115:0/1256197181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe860104c70 msgr2=0x7fe8600ffeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:27.796 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe865785700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe860104c70 0x7fe8600ffeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:27.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe865785700 1 -- 172.21.15.115:0/1256197181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8600febc0 msgr2=0x7fe8600ff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:27.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe865785700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8600febc0 0x7fe8600ff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:27.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe865785700 1 -- 172.21.15.115:0/1256197181 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8540095f0 con 0x7fe8600fe210 2023-07-14T18:11:27.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.748+0000 7fe864f84700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8600febc0 0x7fe8600ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:27.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.749+0000 7fe865785700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 0x7fe8600ff3f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe85800ba50 tx=0x7fe85800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:27.797 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.749+0000 7fe85e7fc700 1 -- 172.21.15.115:0/1256197181 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85800cab0 con 0x7fe8600fe210 2023-07-14T18:11:27.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.749+0000 7fe85e7fc700 1 -- 172.21.15.115:0/1256197181 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe85800cc10 con 0x7fe8600fe210 2023-07-14T18:11:27.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.749+0000 7fe85e7fc700 1 -- 172.21.15.115:0/1256197181 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8580075d0 con 0x7fe8600fe210 2023-07-14T18:11:27.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.749+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8601a4920 con 0x7fe8600fe210 2023-07-14T18:11:27.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.749+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8601a4e20 con 0x7fe8600fe210 2023-07-14T18:11:27.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.750+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86004e340 con 0x7fe8600fe210 2023-07-14T18:11:27.798 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.751+0000 7fe85e7fc700 1 -- 172.21.15.115:0/1256197181 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fe858014440 con 0x7fe8600fe210 2023-07-14T18:11:27.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.752+0000 7fe85e7fc700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8480a1260 0x7fe8480a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:27.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.752+0000 7fe864f84700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8480a1260 0x7fe8480a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:27.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.752+0000 7fe85e7fc700 1 -- 172.21.15.115:0/1256197181 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe858006e40 con 0x7fe8600fe210 2023-07-14T18:11:27.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.752+0000 7fe864f84700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8480a1260 0x7fe8480a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe85400b5c0 tx=0x7fe854005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:27.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.755+0000 7fe85e7fc700 1 -- 172.21.15.115:0/1256197181 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe858096020 con 0x7fe8600fe210 2023-07-14T18:11:28.023 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.984+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe860100920 con 0x7fe8600fe210 2023-07-14T18:11:28.024 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.984+0000 7fe85e7fc700 1 -- 172.21.15.115:0/1256197181 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe858019080 con 0x7fe8600fe210 2023-07-14T18:11:28.024 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:28.024 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-14T18:11:28.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8480a1260 msgr2=0x7fe8480a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8480a1260 0x7fe8480a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe85400b5c0 tx=0x7fe854005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 msgr2=0x7fe8600ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 0x7fe8600ff3f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe85800ba50 tx=0x7fe85800be10 comp rx=0 tx=0).stop 2023-07-14T18:11:28.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 shutdown_connections 2023-07-14T18:11:28.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8480a1260 0x7fe8480a3710 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe860104c70 0x7fe8600ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe8600fe210 0x7fe8600ff3f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 --2- 172.21.15.115:0/1256197181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8600febc0 0x7fe8600ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 >> 172.21.15.115:0/1256197181 conn(0x7fe860075280 msgr2=0x7fe8600fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:28.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 shutdown_connections 2023-07-14T18:11:28.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:27.986+0000 7fe8679e9700 1 -- 172.21.15.115:0/1256197181 wait complete. 2023-07-14T18:11:28.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:28 smithi115 ceph-mon[127451]: pgmap v286: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T18:11:28.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:28 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1256197181' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T18:11:28.520 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-14T18:11:28.520 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:11:28.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:28 smithi150 ceph-mon[128853]: pgmap v286: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T18:11:28.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:28 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1256197181' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T18:11:28.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 -- 172.21.15.115:0/2907196766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501006b0 msgr2=0x7f7850102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 --2- 172.21.15.115:0/2907196766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501006b0 0x7f7850102a90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7844009990 tx=0x7f7844009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 -- 172.21.15.115:0/2907196766 shutdown_connections 2023-07-14T18:11:28.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 --2- 172.21.15.115:0/2907196766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78501058f0 0x7f7850107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 --2- 172.21.15.115:0/2907196766 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501006b0 0x7f7850102a90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 --2- 172.21.15.115:0/2907196766 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7850102fd0 0x7f78501053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 -- 172.21.15.115:0/2907196766 >> 172.21.15.115:0/2907196766 conn(0x7f78500fa150 msgr2=0x7f78500fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:28.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 -- 172.21.15.115:0/2907196766 shutdown_connections 2023-07-14T18:11:28.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 -- 172.21.15.115:0/2907196766 wait complete. 2023-07-14T18:11:28.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 Processor -- start 2023-07-14T18:11:28.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.606+0000 7f785760f700 1 -- start start 2023-07-14T18:11:28.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f785760f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78501006b0 0x7f7850195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:28.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f785760f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7850102fd0 0x7f7850195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:28.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f785760f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501058f0 0x7f785019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:28.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f785760f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785019d3a0 con 0x7f78501058f0 2023-07-14T18:11:28.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f785760f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785019d4e0 con 0x7f78501006b0 2023-07-14T18:11:28.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f785760f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785019d670 con 0x7f7850102fd0 2023-07-14T18:11:28.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7855bac700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501058f0 0x7f785019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:28.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7855bac700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501058f0 0x7f785019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53576/0 (socket says 172.21.15.115:53576) 2023-07-14T18:11:28.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7855bac700 1 -- 172.21.15.115:0/2223434417 learned_addr learned my addr 172.21.15.115:0/2223434417 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:28.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f78553ab700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78501006b0 0x7f7850195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:28.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7854baa700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7850102fd0 0x7f7850195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:28.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7855bac700 1 -- 172.21.15.115:0/2223434417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7850102fd0 msgr2=0x7f7850195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7855bac700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7850102fd0 0x7f7850195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7855bac700 1 -- 172.21.15.115:0/2223434417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78501006b0 msgr2=0x7f7850195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7855bac700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78501006b0 0x7f7850195820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7855bac700 1 -- 172.21.15.115:0/2223434417 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78440095f0 con 0x7f78501058f0 2023-07-14T18:11:28.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f78553ab700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78501006b0 0x7f7850195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:28.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f7855bac700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501058f0 0x7f785019cdd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7848007ee0 tx=0x7f784800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:28.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f78427fc700 1 -- 172.21.15.115:0/2223434417 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7848017070 con 0x7f78501058f0 2023-07-14T18:11:28.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f78427fc700 1 -- 172.21.15.115:0/2223434417 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f784800f040 con 0x7f78501058f0 2023-07-14T18:11:28.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f78427fc700 1 -- 172.21.15.115:0/2223434417 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7848005710 con 0x7f78501058f0 2023-07-14T18:11:28.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785019d950 con 0x7f78501058f0 2023-07-14T18:11:28.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.607+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785019de60 con 0x7f78501058f0 2023-07-14T18:11:28.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.608+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78500fbcc0 con 0x7f78501058f0 2023-07-14T18:11:28.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.609+0000 7f78427fc700 1 -- 172.21.15.115:0/2223434417 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f7848005870 con 0x7f78501058f0 2023-07-14T18:11:28.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.610+0000 7f78427fc700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f783c0a12a0 0x7f783c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:28.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.610+0000 7f78553ab700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f783c0a12a0 0x7f783c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:28.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.610+0000 7f78427fc700 1 -- 172.21.15.115:0/2223434417 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f78480ccad0 con 0x7f78501058f0 2023-07-14T18:11:28.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.610+0000 7f78553ab700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f783c0a12a0 0x7f783c0a3750 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f784400b5c0 tx=0x7f7844005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:28.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.613+0000 7f78427fc700 1 -- 172.21.15.115:0/2223434417 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7848095ed0 con 0x7f78501058f0 2023-07-14T18:11:28.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:28 smithi187 ceph-mon[129317]: pgmap v286: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T18:11:28.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:28 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1256197181' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T18:11:28.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.803+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f785002c8e0 con 0x7f78501058f0 2023-07-14T18:11:28.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.805+0000 7f78427fc700 1 -- 172.21.15.115:0/2223434417 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7f78480994f0 con 0x7f78501058f0 2023-07-14T18:11:28.845 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:28.845 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":87,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:10:22.843435+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:11:28.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f783c0a12a0 msgr2=0x7f783c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f783c0a12a0 0x7f783c0a3750 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f784400b5c0 tx=0x7f7844005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501058f0 msgr2=0x7f785019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501058f0 0x7f785019cdd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7848007ee0 tx=0x7f784800d3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 shutdown_connections 2023-07-14T18:11:28.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f783c0a12a0 0x7f783c0a3750 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f78501058f0 0x7f785019cdd0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78501006b0 0x7f7850195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 --2- 172.21.15.115:0/2223434417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7850102fd0 0x7f7850195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 >> 172.21.15.115:0/2223434417 conn(0x7f78500fa150 msgr2=0x7f7850101310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:28.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 shutdown_connections 2023-07-14T18:11:28.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.807+0000 7f785760f700 1 -- 172.21.15.115:0/2223434417 wait complete. 2023-07-14T18:11:28.859 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:10:15.471213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:10:16.791973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:10:20.663275+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:11:28.860 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:11:28.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.947+0000 7fa0fadde700 1 -- 172.21.15.115:0/1704975368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40feb10 msgr2=0x7fa0f40fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.947+0000 7fa0fadde700 1 --2- 172.21.15.115:0/1704975368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40feb10 0x7fa0f40fef20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009990 tx=0x7fa0e8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.947+0000 7fa0fadde700 1 -- 172.21.15.115:0/1704975368 shutdown_connections 2023-07-14T18:11:28.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.947+0000 7fa0fadde700 1 --2- 172.21.15.115:0/1704975368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0f4100690 0x7fa0f4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.947+0000 7fa0fadde700 1 --2- 172.21.15.115:0/1704975368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40feb10 0x7fa0f40fef20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.947+0000 7fa0fadde700 1 --2- 172.21.15.115:0/1704975368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0f40ffd00 0x7fa0f4100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.947+0000 7fa0fadde700 1 -- 172.21.15.115:0/1704975368 >> 172.21.15.115:0/1704975368 conn(0x7fa0f40fa130 msgr2=0x7fa0f40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:28.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.947+0000 7fa0fadde700 1 -- 172.21.15.115:0/1704975368 shutdown_connections 2023-07-14T18:11:28.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.947+0000 7fa0fadde700 1 -- 172.21.15.115:0/1704975368 wait complete. 2023-07-14T18:11:28.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0fadde700 1 Processor -- start 2023-07-14T18:11:28.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0fadde700 1 -- start start 2023-07-14T18:11:28.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0fadde700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0f40feb10 0x7fa0f4194830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:28.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0fadde700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40ffd00 0x7fa0f4194d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:28.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0fadde700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0f4100690 0x7fa0f419bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:28.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0fadde700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f419c3b0 con 0x7fa0f40ffd00 2023-07-14T18:11:28.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0fadde700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f419c4f0 con 0x7fa0f40feb10 2023-07-14T18:11:28.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0fadde700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f419c680 con 0x7fa0f4100690 2023-07-14T18:11:28.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0f3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40ffd00 0x7fa0f4194d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:28.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0f3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40ffd00 0x7fa0f4194d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53588/0 (socket says 172.21.15.115:53588) 2023-07-14T18:11:28.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0f3fff700 1 -- 172.21.15.115:0/3594131765 learned_addr learned my addr 172.21.15.115:0/3594131765 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:28.991 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.948+0000 7fa0f8b7a700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0f40feb10 0x7fa0f4194830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:28.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f937b700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0f4100690 0x7fa0f419bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:28.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f3fff700 1 -- 172.21.15.115:0/3594131765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0f4100690 msgr2=0x7fa0f419bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f3fff700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0f4100690 0x7fa0f419bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f3fff700 1 -- 172.21.15.115:0/3594131765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0f40feb10 msgr2=0x7fa0f4194830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:28.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f3fff700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0f40feb10 0x7fa0f4194830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:28.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f3fff700 1 -- 172.21.15.115:0/3594131765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e80095f0 con 0x7fa0f40ffd00 2023-07-14T18:11:28.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f3fff700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40ffd00 0x7fa0f4194d70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000d980 tx=0x7fa0e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:28.992 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f1ffb700 1 -- 172.21.15.115:0/3594131765 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0004220 con 0x7fa0f40ffd00 2023-07-14T18:11:28.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f419c960 con 0x7fa0f40ffd00 2023-07-14T18:11:28.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f1ffb700 1 -- 172.21.15.115:0/3594131765 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa0e0007640 con 0x7fa0f40ffd00 2023-07-14T18:11:28.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0f1ffb700 1 -- 172.21.15.115:0/3594131765 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00055e0 con 0x7fa0f40ffd00 2023-07-14T18:11:28.993 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.949+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f419ce70 con 0x7fa0f40ffd00 2023-07-14T18:11:28.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.950+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f410a1e0 con 0x7fa0f40ffd00 2023-07-14T18:11:28.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.951+0000 7fa0f1ffb700 1 -- 172.21.15.115:0/3594131765 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fa0e0004750 con 0x7fa0f40ffd00 2023-07-14T18:11:28.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.952+0000 7fa0f1ffb700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa0e40a11e0 0x7fa0e40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:28.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.952+0000 7fa0f1ffb700 1 -- 172.21.15.115:0/3594131765 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa0e00ccb60 con 0x7fa0f40ffd00 2023-07-14T18:11:28.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.952+0000 7fa0f8b7a700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa0e40a11e0 0x7fa0e40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:28.994 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.952+0000 7fa0f8b7a700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa0e40a11e0 0x7fa0e40a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80098e0 tx=0x7fa0e800b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:28.995 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:28.956+0000 7fa0f1ffb700 1 -- 172.21.15.115:0/3594131765 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa0e0095f00 con 0x7fa0f40ffd00 2023-07-14T18:11:29.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.144+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0f4105af0 con 0x7fa0f40ffd00 2023-07-14T18:11:29.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.144+0000 7fa0f1ffb700 1 -- 172.21.15.115:0/3594131765 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0e0016020 con 0x7fa0f40ffd00 2023-07-14T18:11:29.184 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:11:29.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa0e40a11e0 msgr2=0x7fa0e40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa0e40a11e0 0x7fa0e40a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80098e0 tx=0x7fa0e800b260 comp rx=0 tx=0).stop 2023-07-14T18:11:29.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40ffd00 msgr2=0x7fa0f4194d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40ffd00 0x7fa0f4194d70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000d980 tx=0x7fa0e000dc90 comp rx=0 tx=0).stop 2023-07-14T18:11:29.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 shutdown_connections 2023-07-14T18:11:29.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa0e40a11e0 0x7fa0e40a3690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0f4100690 0x7fa0f419bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0f40feb10 0x7fa0f4194830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 --2- 172.21.15.115:0/3594131765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa0f40ffd00 0x7fa0f4194d70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.146+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 >> 172.21.15.115:0/3594131765 conn(0x7fa0f40fa130 msgr2=0x7fa0f40fc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:29.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.147+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 shutdown_connections 2023-07-14T18:11:29.188 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.147+0000 7fa0fadde700 1 -- 172.21.15.115:0/3594131765 wait complete. 2023-07-14T18:11:29.198 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:11:29.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.287+0000 7ff3b2a76700 1 -- 172.21.15.115:0/4243352181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 msgr2=0x7ff3ac0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.327 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.287+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/4243352181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 0x7ff3ac0fef20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009990 tx=0x7ff3a0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.287+0000 7ff3b2a76700 1 -- 172.21.15.115:0/4243352181 shutdown_connections 2023-07-14T18:11:29.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.287+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/4243352181 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac100690 0x7ff3ac106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.287+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/4243352181 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 0x7ff3ac0fef20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.287+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/4243352181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3ac0ffd00 0x7ff3ac100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.287+0000 7ff3b2a76700 1 -- 172.21.15.115:0/4243352181 >> 172.21.15.115:0/4243352181 conn(0x7ff3ac0fa130 msgr2=0x7ff3ac0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:29.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.288+0000 7ff3b2a76700 1 -- 172.21.15.115:0/4243352181 shutdown_connections 2023-07-14T18:11:29.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.288+0000 7ff3b2a76700 1 -- 172.21.15.115:0/4243352181 wait complete. 2023-07-14T18:11:29.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.288+0000 7ff3b2a76700 1 Processor -- start 2023-07-14T18:11:29.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.288+0000 7ff3b2a76700 1 -- start start 2023-07-14T18:11:29.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.288+0000 7ff3b2a76700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 0x7ff3ac1947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:29.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b2a76700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3ac0ffd00 0x7ff3ac194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:29.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b2a76700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac100690 0x7ff3ac19bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:29.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b2a76700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac19c370 con 0x7ff3ac0feb10 2023-07-14T18:11:29.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b2a76700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac19c4b0 con 0x7ff3ac0ffd00 2023-07-14T18:11:29.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b2a76700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac19c640 con 0x7ff3ac100690 2023-07-14T18:11:29.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b0812700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 0x7ff3ac1947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:29.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b0812700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 0x7ff3ac1947f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53600/0 (socket says 172.21.15.115:53600) 2023-07-14T18:11:29.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b0812700 1 -- 172.21.15.115:0/1764838139 learned_addr learned my addr 172.21.15.115:0/1764838139 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:29.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3abfff700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3ac0ffd00 0x7ff3ac194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:29.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b0812700 1 -- 172.21.15.115:0/1764838139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac100690 msgr2=0x7ff3ac19bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b0812700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac100690 0x7ff3ac19bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b0812700 1 -- 172.21.15.115:0/1764838139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3ac0ffd00 msgr2=0x7ff3ac194d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b0812700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3ac0ffd00 0x7ff3ac194d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.332 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b0812700 1 -- 172.21.15.115:0/1764838139 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a00095f0 con 0x7ff3ac0feb10 2023-07-14T18:11:29.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3abfff700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3ac0ffd00 0x7ff3ac194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:29.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b0812700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 0x7ff3ac1947f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0004730 tx=0x7ff3a0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:29.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3a9ffb700 1 -- 172.21.15.115:0/1764838139 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a001d070 con 0x7ff3ac0feb10 2023-07-14T18:11:29.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3a9ffb700 1 -- 172.21.15.115:0/1764838139 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff3a0004080 con 0x7ff3ac0feb10 2023-07-14T18:11:29.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac19c8c0 con 0x7ff3ac0feb10 2023-07-14T18:11:29.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3a9ffb700 1 -- 172.21.15.115:0/1764838139 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0004ea0 con 0x7ff3ac0feb10 2023-07-14T18:11:29.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.289+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac19cce0 con 0x7ff3ac0feb10 2023-07-14T18:11:29.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.290+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac105af0 con 0x7ff3ac0feb10 2023-07-14T18:11:29.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.291+0000 7ff3a9ffb700 1 -- 172.21.15.115:0/1764838139 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff3a0003bf0 con 0x7ff3ac0feb10 2023-07-14T18:11:29.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.292+0000 7ff3a9ffb700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff39c0a11e0 0x7ff39c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:29.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.292+0000 7ff3a9ffb700 1 -- 172.21.15.115:0/1764838139 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff3a00ce230 con 0x7ff3ac0feb10 2023-07-14T18:11:29.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.292+0000 7ff3abfff700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff39c0a11e0 0x7ff39c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:29.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.292+0000 7ff3abfff700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff39c0a11e0 0x7ff39c0a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff398005fd0 tx=0x7ff398005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:29.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.295+0000 7ff3a9ffb700 1 -- 172.21.15.115:0/1764838139 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff3a0097540 con 0x7ff3ac0feb10 2023-07-14T18:11:29.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.484+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff3ac02ca70 con 0x7ff3ac0feb10 2023-07-14T18:11:29.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.484+0000 7ff3a9ffb700 1 -- 172.21.15.115:0/1764838139 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff3a009ab60 con 0x7ff3ac0feb10 2023-07-14T18:11:29.524 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:11:29.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff39c0a11e0 msgr2=0x7ff39c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff39c0a11e0 0x7ff39c0a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff398005fd0 tx=0x7ff398005ee0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 msgr2=0x7ff3ac1947f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 0x7ff3ac1947f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0004730 tx=0x7ff3a0009580 comp rx=0 tx=0).stop 2023-07-14T18:11:29.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 shutdown_connections 2023-07-14T18:11:29.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff39c0a11e0 0x7ff39c0a3690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3ac100690 0x7ff3ac19bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff3ac0feb10 0x7ff3ac1947f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 --2- 172.21.15.115:0/1764838139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3ac0ffd00 0x7ff3ac194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 >> 172.21.15.115:0/1764838139 conn(0x7ff3ac0fa130 msgr2=0x7ff3ac0fc490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:29.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 shutdown_connections 2023-07-14T18:11:29.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.486+0000 7ff3b2a76700 1 -- 172.21.15.115:0/1764838139 wait complete. 2023-07-14T18:11:29.537 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2223434417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2023-07-14T18:11:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2023-07-14T18:11:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-07-14T18:11:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2023-07-14T18:11:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-07-14T18:11:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-07-14T18:11:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2023-07-14T18:11:29.538 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-07-14T18:11:29.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-07-14T18:11:29.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 10]}]: dispatch 2023-07-14T18:11:29.539 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:29 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3594131765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:29.540 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:11:29.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2223434417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:29.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2023-07-14T18:11:29.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2023-07-14T18:11:29.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-07-14T18:11:29.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2023-07-14T18:11:29.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-07-14T18:11:29.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-07-14T18:11:29.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2023-07-14T18:11:29.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-07-14T18:11:29.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-07-14T18:11:29.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 10]}]: dispatch 2023-07-14T18:11:29.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:29 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3594131765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:29.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 -- 172.21.15.115:0/2855570477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c0feb10 msgr2=0x7ff44c0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 --2- 172.21.15.115:0/2855570477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c0feb10 0x7ff44c0fef20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff448009990 tx=0x7ff448009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 -- 172.21.15.115:0/2855570477 shutdown_connections 2023-07-14T18:11:29.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 --2- 172.21.15.115:0/2855570477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c100690 0x7ff44c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.664 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 --2- 172.21.15.115:0/2855570477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c0feb10 0x7ff44c0fef20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 --2- 172.21.15.115:0/2855570477 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff44c0ffd00 0x7ff44c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 -- 172.21.15.115:0/2855570477 >> 172.21.15.115:0/2855570477 conn(0x7ff44c0fa130 msgr2=0x7ff44c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:29.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 -- 172.21.15.115:0/2855570477 shutdown_connections 2023-07-14T18:11:29.665 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 -- 172.21.15.115:0/2855570477 wait complete. 2023-07-14T18:11:29.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 Processor -- start 2023-07-14T18:11:29.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.624+0000 7ff4544a7700 1 -- start start 2023-07-14T18:11:29.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff4544a7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff44c0feb10 0x7ff44c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:29.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff4544a7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c0ffd00 0x7ff44c107a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:29.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff4544a7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c100690 0x7ff44c107f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:29.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff4544a7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c108720 con 0x7ff44c100690 2023-07-14T18:11:29.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff4544a7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c108890 con 0x7ff44c0ffd00 2023-07-14T18:11:29.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff4544a7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c19f9b0 con 0x7ff44c0feb10 2023-07-14T18:11:29.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452a44700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c100690 0x7ff44c107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:29.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452a44700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c100690 0x7ff44c107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53618/0 (socket says 172.21.15.115:53618) 2023-07-14T18:11:29.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452a44700 1 -- 172.21.15.115:0/3797087179 learned_addr learned my addr 172.21.15.115:0/3797087179 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:29.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452243700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff44c0feb10 0x7ff44c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:29.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff451a42700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c0ffd00 0x7ff44c107a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:29.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452a44700 1 -- 172.21.15.115:0/3797087179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff44c0feb10 msgr2=0x7ff44c109400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452a44700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff44c0feb10 0x7ff44c109400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452a44700 1 -- 172.21.15.115:0/3797087179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c0ffd00 msgr2=0x7ff44c107a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452a44700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c0ffd00 0x7ff44c107a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452a44700 1 -- 172.21.15.115:0/3797087179 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4480095f0 con 0x7ff44c100690 2023-07-14T18:11:29.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff451a42700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c0ffd00 0x7ff44c107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:29.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff452a44700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c100690 0x7ff44c107f80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff44400b970 tx=0x7ff44400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:29.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff43f7fe700 1 -- 172.21.15.115:0/3797087179 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44400c930 con 0x7ff44c100690 2023-07-14T18:11:29.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff43f7fe700 1 -- 172.21.15.115:0/3797087179 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff444014440 con 0x7ff44c100690 2023-07-14T18:11:29.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44c19fc90 con 0x7ff44c100690 2023-07-14T18:11:29.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff43f7fe700 1 -- 172.21.15.115:0/3797087179 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4440075d0 con 0x7ff44c100690 2023-07-14T18:11:29.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.625+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44c1a0150 con 0x7ff44c100690 2023-07-14T18:11:29.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.626+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44c105af0 con 0x7ff44c100690 2023-07-14T18:11:29.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.627+0000 7ff43f7fe700 1 -- 172.21.15.115:0/3797087179 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff4440145b0 con 0x7ff44c100690 2023-07-14T18:11:29.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.628+0000 7ff43f7fe700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4380a11e0 0x7ff4380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:29.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.628+0000 7ff452243700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4380a11e0 0x7ff4380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:29.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.628+0000 7ff43f7fe700 1 -- 172.21.15.115:0/3797087179 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff444006e40 con 0x7ff44c100690 2023-07-14T18:11:29.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.628+0000 7ff452243700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4380a11e0 0x7ff4380a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff44800c010 tx=0x7ff448005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:29.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.631+0000 7ff43f7fe700 1 -- 172.21.15.115:0/3797087179 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff444096030 con 0x7ff44c100690 2023-07-14T18:11:29.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2223434417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:29.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2023-07-14T18:11:29.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2023-07-14T18:11:29.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-07-14T18:11:29.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2023-07-14T18:11:29.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-07-14T18:11:29.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-07-14T18:11:29.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2023-07-14T18:11:29.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-07-14T18:11:29.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-07-14T18:11:29.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 10]}]: dispatch 2023-07-14T18:11:29.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:29 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3594131765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:29.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.819+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff44c02ca70 con 0x7ff44c100690 2023-07-14T18:11:29.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.819+0000 7ff43f7fe700 1 -- 172.21.15.115:0/3797087179 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff4440190a0 con 0x7ff44c100690 2023-07-14T18:11:29.859 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 64 2023-07-14T18:11:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4380a11e0 msgr2=0x7ff4380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4380a11e0 0x7ff4380a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff44800c010 tx=0x7ff448005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c100690 msgr2=0x7ff44c107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c100690 0x7ff44c107f80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff44400b970 tx=0x7ff44400bc80 comp rx=0 tx=0).stop 2023-07-14T18:11:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 shutdown_connections 2023-07-14T18:11:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff4380a11e0 0x7ff4380a3690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff44c100690 0x7ff44c107f80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff44c0feb10 0x7ff44c109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 --2- 172.21.15.115:0/3797087179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c0ffd00 0x7ff44c107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 >> 172.21.15.115:0/3797087179 conn(0x7ff44c0fa130 msgr2=0x7ff44c0fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 shutdown_connections 2023-07-14T18:11:29.862 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.821+0000 7ff4544a7700 1 -- 172.21.15.115:0/3797087179 wait complete. 2023-07-14T18:11:29.873 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:11:29.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.958+0000 7f525b951700 1 -- 172.21.15.115:0/3934725597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 msgr2=0x7f52540ff1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:29.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.958+0000 7f525b951700 1 --2- 172.21.15.115:0/3934725597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 0x7f52540ff1d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5250009990 tx=0x7f5250009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.958+0000 7f525b951700 1 -- 172.21.15.115:0/3934725597 shutdown_connections 2023-07-14T18:11:29.998 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.958+0000 7f525b951700 1 --2- 172.21.15.115:0/3934725597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52540ff710 0x7f5254106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.958+0000 7f525b951700 1 --2- 172.21.15.115:0/3934725597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5254100110 0x7f5254100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.958+0000 7f525b951700 1 --2- 172.21.15.115:0/3934725597 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 0x7f52540ff1d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:29.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.958+0000 7f525b951700 1 -- 172.21.15.115:0/3934725597 >> 172.21.15.115:0/3934725597 conn(0x7f52540f98a0 msgr2=0x7f52540fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:29.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.958+0000 7f525b951700 1 -- 172.21.15.115:0/3934725597 shutdown_connections 2023-07-14T18:11:29.999 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.958+0000 7f525b951700 1 -- 172.21.15.115:0/3934725597 wait complete. 2023-07-14T18:11:30.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 Processor -- start 2023-07-14T18:11:30.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 -- start start 2023-07-14T18:11:30.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 0x7f5254102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52540ff710 0x7f5254101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5254100110 0x7f5254101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.001 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5254101e30 con 0x7f52540fed60 2023-07-14T18:11:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5254101fa0 con 0x7f52540ff710 2023-07-14T18:11:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525419f930 con 0x7f5254100110 2023-07-14T18:11:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f52596ed700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 0x7f5254102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f52596ed700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 0x7f5254102b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53628/0 (socket says 172.21.15.115:53628) 2023-07-14T18:11:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f52596ed700 1 -- 172.21.15.115:0/3060995047 learned_addr learned my addr 172.21.15.115:0/3060995047 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f52596ed700 1 -- 172.21.15.115:0/3060995047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5254100110 msgr2=0x7f5254101690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.002 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f52596ed700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5254100110 0x7f5254101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f52596ed700 1 -- 172.21.15.115:0/3060995047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52540ff710 msgr2=0x7f5254101150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f52596ed700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52540ff710 0x7f5254101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f52596ed700 1 -- 172.21.15.115:0/3060995047 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500095f0 con 0x7f52540fed60 2023-07-14T18:11:30.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f52596ed700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 0x7f5254102b10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f524400c970 tx=0x7f524400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:30.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f524a7fc700 1 -- 172.21.15.115:0/3060995047 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244004220 con 0x7f52540fed60 2023-07-14T18:11:30.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525419fc10 con 0x7f52540fed60 2023-07-14T18:11:30.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f524a7fc700 1 -- 172.21.15.115:0/3060995047 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5244007d30 con 0x7f52540fed60 2023-07-14T18:11:30.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f524a7fc700 1 -- 172.21.15.115:0/3060995047 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244003bf0 con 0x7f52540fed60 2023-07-14T18:11:30.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.959+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52541a01a0 con 0x7f52540fed60 2023-07-14T18:11:30.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.960+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525410a200 con 0x7f52540fed60 2023-07-14T18:11:30.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.961+0000 7f524a7fc700 1 -- 172.21.15.115:0/3060995047 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f52440078a0 con 0x7f52540fed60 2023-07-14T18:11:30.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.962+0000 7f524a7fc700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52400a11e0 0x7f52400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.962+0000 7f5258eec700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52400a11e0 0x7f52400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.962+0000 7f524a7fc700 1 -- 172.21.15.115:0/3060995047 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f52440cca40 con 0x7f52540fed60 2023-07-14T18:11:30.005 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.962+0000 7f5258eec700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52400a11e0 0x7f52400a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f52500098e0 tx=0x7f5250000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:30.006 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:29.966+0000 7f524a7fc700 1 -- 172.21.15.115:0/3060995047 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5244095d20 con 0x7f52540fed60 2023-07-14T18:11:30.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.158+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f52541a07b0 con 0x7f52540fed60 2023-07-14T18:11:30.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.158+0000 7f524a7fc700 1 -- 172.21.15.115:0/3060995047 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f524401d0a0 con 0x7f52540fed60 2023-07-14T18:11:30.198 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:11:30.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52400a11e0 msgr2=0x7f52400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52400a11e0 0x7f52400a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f52500098e0 tx=0x7f5250000bc0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 msgr2=0x7f5254102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 0x7f5254102b10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f524400c970 tx=0x7f524400cd30 comp rx=0 tx=0).stop 2023-07-14T18:11:30.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 shutdown_connections 2023-07-14T18:11:30.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f52400a11e0 0x7f52400a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5254100110 0x7f5254101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f52540fed60 0x7f5254102b10 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 --2- 172.21.15.115:0/3060995047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52540ff710 0x7f5254101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 >> 172.21.15.115:0/3060995047 conn(0x7f52540f98a0 msgr2=0x7f52540fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:30.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 shutdown_connections 2023-07-14T18:11:30.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.160+0000 7f525b951700 1 -- 172.21.15.115:0/3060995047 wait complete. 2023-07-14T18:11:30.212 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:11:30.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 -- 172.21.15.115:0/2603078002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c0feb00 msgr2=0x7f830c0fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 --2- 172.21.15.115:0/2603078002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c0feb00 0x7f830c0fef10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8300009990 tx=0x7f8300009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 -- 172.21.15.115:0/2603078002 shutdown_connections 2023-07-14T18:11:30.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 --2- 172.21.15.115:0/2603078002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f830c100680 0x7f830c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 --2- 172.21.15.115:0/2603078002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c0feb00 0x7f830c0fef10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 --2- 172.21.15.115:0/2603078002 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f830c0ffcf0 0x7f830c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 -- 172.21.15.115:0/2603078002 >> 172.21.15.115:0/2603078002 conn(0x7f830c0fa100 msgr2=0x7f830c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:30.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 -- 172.21.15.115:0/2603078002 shutdown_connections 2023-07-14T18:11:30.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 -- 172.21.15.115:0/2603078002 wait complete. 2023-07-14T18:11:30.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 Processor -- start 2023-07-14T18:11:30.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.298+0000 7f8312a07700 1 -- start start 2023-07-14T18:11:30.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8312a07700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f830c0feb00 0x7f830c068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8312a07700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f830c0ffcf0 0x7f830c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8312a07700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c100680 0x7f830c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.341 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8312a07700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c19d370 con 0x7f830c100680 2023-07-14T18:11:30.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8312a07700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c19d4e0 con 0x7f830c0feb00 2023-07-14T18:11:30.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8312a07700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c19d6a0 con 0x7f830c0ffcf0 2023-07-14T18:11:30.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8310fa4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c100680 0x7f830c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8310fa4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c100680 0x7f830c19cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53646/0 (socket says 172.21.15.115:53646) 2023-07-14T18:11:30.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8310fa4700 1 -- 172.21.15.115:0/1638229723 learned_addr learned my addr 172.21.15.115:0/1638229723 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:30.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f830bfff700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f830c0feb00 0x7f830c068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f830b7fe700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f830c0ffcf0 0x7f830c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.342 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8310fa4700 1 -- 172.21.15.115:0/1638229723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f830c0ffcf0 msgr2=0x7f830c19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8310fa4700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f830c0ffcf0 0x7f830c19a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8310fa4700 1 -- 172.21.15.115:0/1638229723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f830c0feb00 msgr2=0x7f830c068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8310fa4700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f830c0feb00 0x7f830c068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8310fa4700 1 -- 172.21.15.115:0/1638229723 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83000095f0 con 0x7f830c100680 2023-07-14T18:11:30.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.299+0000 7f8310fa4700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c100680 0x7f830c19cba0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8304007ee0 tx=0x7f830400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:30.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.300+0000 7f83097fa700 1 -- 172.21.15.115:0/1638229723 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304017070 con 0x7f830c100680 2023-07-14T18:11:30.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.300+0000 7f83097fa700 1 -- 172.21.15.115:0/1638229723 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f830400f040 con 0x7f830c100680 2023-07-14T18:11:30.343 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.300+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830c19d980 con 0x7f830c100680 2023-07-14T18:11:30.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.300+0000 7f83097fa700 1 -- 172.21.15.115:0/1638229723 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304005710 con 0x7f830c100680 2023-07-14T18:11:30.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.300+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830c19de40 con 0x7f830c100680 2023-07-14T18:11:30.344 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.301+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c105ab0 con 0x7f830c100680 2023-07-14T18:11:30.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.302+0000 7f83097fa700 1 -- 172.21.15.115:0/1638229723 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f8304005870 con 0x7f830c100680 2023-07-14T18:11:30.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.302+0000 7f83097fa700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82f40a1220 0x7f82f40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.302+0000 7f830bfff700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82f40a1220 0x7f82f40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.302+0000 7f83097fa700 1 -- 172.21.15.115:0/1638229723 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f83040ccbc0 con 0x7f830c100680 2023-07-14T18:11:30.345 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.303+0000 7f830bfff700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82f40a1220 0x7f82f40a36d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f830000b5c0 tx=0x7f8300005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:30.346 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.305+0000 7f83097fa700 1 -- 172.21.15.115:0/1638229723 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8304095ea0 con 0x7f830c100680 2023-07-14T18:11:30.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.537+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f830c02c8e0 con 0x7f830c100680 2023-07-14T18:11:30.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.537+0000 7f83097fa700 1 -- 172.21.15.115:0/1638229723 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10851 (secure 0 0 0) 0x7f83040994c0 con 0x7f830c100680 2023-07-14T18:11:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:30.577 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:10:38.422616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24493,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24446":{"gid":24446,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:11:30.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.539+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82f40a1220 msgr2=0x7f82f40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.539+0000 7f8312a07700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82f40a1220 0x7f82f40a36d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f830000b5c0 tx=0x7f8300005960 comp rx=0 tx=0).stop 2023-07-14T18:11:30.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.539+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c100680 msgr2=0x7f830c19cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.539+0000 7f8312a07700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c100680 0x7f830c19cba0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8304007ee0 tx=0x7f830400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.539+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 shutdown_connections 2023-07-14T18:11:30.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.539+0000 7f8312a07700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82f40a1220 0x7f82f40a36d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.540+0000 7f8312a07700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f830c100680 0x7f830c19cba0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.540+0000 7f8312a07700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f830c0feb00 0x7f830c068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.540+0000 7f8312a07700 1 --2- 172.21.15.115:0/1638229723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f830c0ffcf0 0x7f830c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.540+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 >> 172.21.15.115:0/1638229723 conn(0x7f830c0fa100 msgr2=0x7f830c0fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:30.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.540+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 shutdown_connections 2023-07-14T18:11:30.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.540+0000 7f8312a07700 1 -- 172.21.15.115:0/1638229723 wait complete. 2023-07-14T18:11:30.581 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 24 2023-07-14T18:11:30.591 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: pgmap v287: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-07-14T18:11:30.591 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2023-07-14T18:11:30.591 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2023-07-14T18:11:30.591 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-07-14T18:11:30.591 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2023-07-14T18:11:30.591 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-07-14T18:11:30.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-07-14T18:11:30.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2023-07-14T18:11:30.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-07-14T18:11:30.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-07-14T18:11:30.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 10]}]': finished 2023-07-14T18:11:30.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T18:11:30.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1764838139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:30.592 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3797087179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:30.593 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:30 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3060995047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:30.594 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:11:30.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: pgmap v287: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-07-14T18:11:30.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2023-07-14T18:11:30.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2023-07-14T18:11:30.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-07-14T18:11:30.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2023-07-14T18:11:30.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-07-14T18:11:30.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-07-14T18:11:30.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2023-07-14T18:11:30.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-07-14T18:11:30.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-07-14T18:11:30.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 10]}]': finished 2023-07-14T18:11:30.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T18:11:30.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1764838139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:30.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3797087179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:30.624 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:30 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3060995047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:30.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: pgmap v287: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-07-14T18:11:30.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2023-07-14T18:11:30.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2023-07-14T18:11:30.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-07-14T18:11:30.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2023-07-14T18:11:30.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-07-14T18:11:30.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-07-14T18:11:30.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2023-07-14T18:11:30.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-07-14T18:11:30.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-07-14T18:11:30.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 10]}]': finished 2023-07-14T18:11:30.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T18:11:30.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1764838139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:30.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3797087179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:30.686 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:30 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3060995047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:30.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.674+0000 7f82590d0700 1 -- 172.21.15.115:0/459308592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 msgr2=0x7f82540feeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.674+0000 7f82590d0700 1 --2- 172.21.15.115:0/459308592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 0x7f82540feeb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8244009990 tx=0x7f8244009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.674+0000 7f82590d0700 1 -- 172.21.15.115:0/459308592 shutdown_connections 2023-07-14T18:11:30.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.674+0000 7f82590d0700 1 --2- 172.21.15.115:0/459308592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8254100620 0x7f8254106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.674+0000 7f82590d0700 1 --2- 172.21.15.115:0/459308592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 0x7f82540feeb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.674+0000 7f82590d0700 1 --2- 172.21.15.115:0/459308592 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f82540ffc90 0x7f82541000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.674+0000 7f82590d0700 1 -- 172.21.15.115:0/459308592 >> 172.21.15.115:0/459308592 conn(0x7f82540fa0c0 msgr2=0x7f82540fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:30.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.674+0000 7f82590d0700 1 -- 172.21.15.115:0/459308592 shutdown_connections 2023-07-14T18:11:30.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.674+0000 7f82590d0700 1 -- 172.21.15.115:0/459308592 wait complete. 2023-07-14T18:11:30.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f82590d0700 1 Processor -- start 2023-07-14T18:11:30.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f82590d0700 1 -- start start 2023-07-14T18:11:30.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f82590d0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 0x7f82541958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f82590d0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82540ffc90 0x7f8254195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f82590d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8254100620 0x7f825419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f82590d0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825419d430 con 0x7f82540feaa0 2023-07-14T18:11:30.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f82590d0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825419d570 con 0x7f82540ffc90 2023-07-14T18:11:30.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f82590d0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825419d700 con 0x7f8254100620 2023-07-14T18:11:30.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f8252d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 0x7f82541958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f8252d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 0x7f82541958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53654/0 (socket says 172.21.15.115:53654) 2023-07-14T18:11:30.718 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.675+0000 7f8252d9d700 1 -- 172.21.15.115:0/2338725320 learned_addr learned my addr 172.21.15.115:0/2338725320 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:30.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f825259c700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82540ffc90 0x7f8254195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f825359e700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8254100620 0x7f825419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f8252d9d700 1 -- 172.21.15.115:0/2338725320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8254100620 msgr2=0x7f825419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f8252d9d700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8254100620 0x7f825419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f8252d9d700 1 -- 172.21.15.115:0/2338725320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82540ffc90 msgr2=0x7f8254195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f8252d9d700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82540ffc90 0x7f8254195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.719 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f8252d9d700 1 -- 172.21.15.115:0/2338725320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440095f0 con 0x7f82540feaa0 2023-07-14T18:11:30.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f825359e700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8254100620 0x7f825419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:30.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f825259c700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82540ffc90 0x7f8254195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:30.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f8252d9d700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 0x7f82541958b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8244004730 tx=0x7f8244009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:30.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f823bfff700 1 -- 172.21.15.115:0/2338725320 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824401d070 con 0x7f82540feaa0 2023-07-14T18:11:30.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f823bfff700 1 -- 172.21.15.115:0/2338725320 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8244004080 con 0x7f82540feaa0 2023-07-14T18:11:30.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825419d980 con 0x7f82540feaa0 2023-07-14T18:11:30.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f823bfff700 1 -- 172.21.15.115:0/2338725320 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244004ea0 con 0x7f82540feaa0 2023-07-14T18:11:30.720 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.676+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f825419de20 con 0x7f82540feaa0 2023-07-14T18:11:30.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.677+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8254106c20 con 0x7f82540feaa0 2023-07-14T18:11:30.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.678+0000 7f823bfff700 1 -- 172.21.15.115:0/2338725320 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f8244003bf0 con 0x7f82540feaa0 2023-07-14T18:11:30.721 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.679+0000 7f823bfff700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f823c0a1220 0x7f823c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:30.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.679+0000 7f825259c700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f823c0a1220 0x7f823c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:30.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.679+0000 7f823bfff700 1 -- 172.21.15.115:0/2338725320 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f82440ce230 con 0x7f82540feaa0 2023-07-14T18:11:30.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.679+0000 7f825259c700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f823c0a1220 0x7f823c0a36d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f82480099d0 tx=0x7f8248008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:30.722 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.682+0000 7f823bfff700 1 -- 172.21.15.115:0/2338725320 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8244097540 con 0x7f82540feaa0 2023-07-14T18:11:30.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.876+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f825402c8e0 con 0x7f82540feaa0 2023-07-14T18:11:30.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.878+0000 7f823bfff700 1 -- 172.21.15.115:0/2338725320 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24558 (secure 0 0 0) 0x7f824409ab60 con 0x7f82540feaa0 2023-07-14T18:11:30.918 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:30.918 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":89,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:11:30.172466+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:11:30.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f823c0a1220 msgr2=0x7f823c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f823c0a1220 0x7f823c0a36d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f82480099d0 tx=0x7f8248008040 comp rx=0 tx=0).stop 2023-07-14T18:11:30.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 msgr2=0x7f82541958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:30.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 0x7f82541958b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8244004730 tx=0x7f8244009580 comp rx=0 tx=0).stop 2023-07-14T18:11:30.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 shutdown_connections 2023-07-14T18:11:30.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f823c0a1220 0x7f823c0a36d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8254100620 0x7f825419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f82540feaa0 0x7f82541958b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 --2- 172.21.15.115:0/2338725320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82540ffc90 0x7f8254195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:30.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 >> 172.21.15.115:0/2338725320 conn(0x7f82540fa0c0 msgr2=0x7f82540fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:30.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 shutdown_connections 2023-07-14T18:11:30.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:30.880+0000 7f82590d0700 1 -- 172.21.15.115:0/2338725320 wait complete. 2023-07-14T18:11:30.933 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:10:15.471213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:10:16.791973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:10:20.663275+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:11:30.934 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:11:31.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 -- 172.21.15.115:0/2854927765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 msgr2=0x7fbe34106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/2854927765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 0x7fbe34106fa0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009990 tx=0x7fbe24009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.060 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 -- 172.21.15.115:0/2854927765 shutdown_connections 2023-07-14T18:11:31.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/2854927765 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 0x7fbe34106fa0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/2854927765 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe34100180 0x7fbe34100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/2854927765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe340fed40 0x7fbe340ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 -- 172.21.15.115:0/2854927765 >> 172.21.15.115:0/2854927765 conn(0x7fbe340f98a0 msgr2=0x7fbe340fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:31.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 -- 172.21.15.115:0/2854927765 shutdown_connections 2023-07-14T18:11:31.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 -- 172.21.15.115:0/2854927765 wait complete. 2023-07-14T18:11:31.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.020+0000 7fbe3af7d700 1 Processor -- start 2023-07-14T18:11:31.061 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3af7d700 1 -- start start 2023-07-14T18:11:31.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3af7d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe340fed40 0x7fbe340fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3af7d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 0x7fbe3419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.063 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3af7d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe34100180 0x7fbe3419cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3af7d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3419d2c0 con 0x7fbe340ff6f0 2023-07-14T18:11:31.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3af7d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3419d430 con 0x7fbe34100180 2023-07-14T18:11:31.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3af7d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3419d5f0 con 0x7fbe340fed40 2023-07-14T18:11:31.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe33fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 0x7fbe3419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe33fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 0x7fbe3419a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53670/0 (socket says 172.21.15.115:53670) 2023-07-14T18:11:31.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe33fff700 1 -- 172.21.15.115:0/392429390 learned_addr learned my addr 172.21.15.115:0/392429390 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:31.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3951a700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe34100180 0x7fbe3419cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe38d19700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe340fed40 0x7fbe340fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe33fff700 1 -- 172.21.15.115:0/392429390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe340fed40 msgr2=0x7fbe340fe7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe33fff700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe340fed40 0x7fbe340fe7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe33fff700 1 -- 172.21.15.115:0/392429390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe34100180 msgr2=0x7fbe3419cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe33fff700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe34100180 0x7fbe3419cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe33fff700 1 -- 172.21.15.115:0/392429390 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe240095f0 con 0x7fbe340ff6f0 2023-07-14T18:11:31.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3951a700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe34100180 0x7fbe3419cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:31.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe33fff700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 0x7fbe3419a4e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00c9b0 tx=0x7fbe1c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:31.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe31ffb700 1 -- 172.21.15.115:0/392429390 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c007b00 con 0x7fbe340ff6f0 2023-07-14T18:11:31.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe31ffb700 1 -- 172.21.15.115:0/392429390 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbe1c007c60 con 0x7fbe340ff6f0 2023-07-14T18:11:31.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3419d8d0 con 0x7fbe340ff6f0 2023-07-14T18:11:31.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.021+0000 7fbe31ffb700 1 -- 172.21.15.115:0/392429390 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c018660 con 0x7fbe340ff6f0 2023-07-14T18:11:31.066 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.022+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3419dde0 con 0x7fbe340ff6f0 2023-07-14T18:11:31.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.023+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3410a1d0 con 0x7fbe340ff6f0 2023-07-14T18:11:31.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.023+0000 7fbe31ffb700 1 -- 172.21.15.115:0/392429390 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fbe1c00f450 con 0x7fbe340ff6f0 2023-07-14T18:11:31.067 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.024+0000 7fbe31ffb700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbe200a11e0 0x7fbe200a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.024+0000 7fbe38d19700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbe200a11e0 0x7fbe200a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.024+0000 7fbe31ffb700 1 -- 172.21.15.115:0/392429390 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbe1c0cced0 con 0x7fbe340ff6f0 2023-07-14T18:11:31.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.024+0000 7fbe38d19700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbe200a11e0 0x7fbe200a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009380 tx=0x7fbe2c00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:31.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.028+0000 7fbe31ffb700 1 -- 172.21.15.115:0/392429390 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbe1c096150 con 0x7fbe340ff6f0 2023-07-14T18:11:31.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.215+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe3410a390 con 0x7fbe340ff6f0 2023-07-14T18:11:31.255 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.216+0000 7fbe31ffb700 1 -- 172.21.15.115:0/392429390 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe1c099770 con 0x7fbe340ff6f0 2023-07-14T18:11:31.256 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:11:31.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbe200a11e0 msgr2=0x7fbe200a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbe200a11e0 0x7fbe200a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009380 tx=0x7fbe2c00b040 comp rx=0 tx=0).stop 2023-07-14T18:11:31.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 msgr2=0x7fbe3419a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 0x7fbe3419a4e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c00c9b0 tx=0x7fbe1c00cd70 comp rx=0 tx=0).stop 2023-07-14T18:11:31.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 shutdown_connections 2023-07-14T18:11:31.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbe200a11e0 0x7fbe200a3690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe34100180 0x7fbe3419cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbe340fed40 0x7fbe340fe7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 --2- 172.21.15.115:0/392429390 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbe340ff6f0 0x7fbe3419a4e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 >> 172.21.15.115:0/392429390 conn(0x7fbe340f98a0 msgr2=0x7fbe340fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:31.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 shutdown_connections 2023-07-14T18:11:31.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.218+0000 7fbe3af7d700 1 -- 172.21.15.115:0/392429390 wait complete. 2023-07-14T18:11:31.270 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:11:31.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.354+0000 7fdfdc46b700 1 -- 172.21.15.115:0/2401723894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 msgr2=0x7fdfd4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.354+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/2401723894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 0x7fdfd4100150 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009990 tx=0x7fdfc4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 -- 172.21.15.115:0/2401723894 shutdown_connections 2023-07-14T18:11:31.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/2401723894 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd4100690 0x7fdfd4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/2401723894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfd40feb10 0x7fdfd40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/2401723894 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 0x7fdfd4100150 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 -- 172.21.15.115:0/2401723894 >> 172.21.15.115:0/2401723894 conn(0x7fdfd40fa130 msgr2=0x7fdfd40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:31.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 -- 172.21.15.115:0/2401723894 shutdown_connections 2023-07-14T18:11:31.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 -- 172.21.15.115:0/2401723894 wait complete. 2023-07-14T18:11:31.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 Processor -- start 2023-07-14T18:11:31.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 -- start start 2023-07-14T18:11:31.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfd40feb10 0x7fdfd4193710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 0x7fdfd4193c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd4100690 0x7fdfd419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd419b290 con 0x7fdfd40ffd00 2023-07-14T18:11:31.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd419b3d0 con 0x7fdfd40feb10 2023-07-14T18:11:31.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfdc46b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd419b560 con 0x7fdfd4100690 2023-07-14T18:11:31.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.355+0000 7fdfd9a06700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 0x7fdfd4193c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfd9a06700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 0x7fdfd4193c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53684/0 (socket says 172.21.15.115:53684) 2023-07-14T18:11:31.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfd9a06700 1 -- 172.21.15.115:0/988148668 learned_addr learned my addr 172.21.15.115:0/988148668 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:31.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfdaa08700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd4100690 0x7fdfd419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfd9a06700 1 -- 172.21.15.115:0/988148668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd4100690 msgr2=0x7fdfd419acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfda207700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfd40feb10 0x7fdfd4193710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfd9a06700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd4100690 0x7fdfd419acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfd9a06700 1 -- 172.21.15.115:0/988148668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfd40feb10 msgr2=0x7fdfd4193710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.399 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfd9a06700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfd40feb10 0x7fdfd4193710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfd9a06700 1 -- 172.21.15.115:0/988148668 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc40095f0 con 0x7fdfd40ffd00 2023-07-14T18:11:31.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfd9a06700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 0x7fdfd4193c50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4003a50 tx=0x7fdfc4003b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:31.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfcb7fe700 1 -- 172.21.15.115:0/988148668 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc401d070 con 0x7fdfd40ffd00 2023-07-14T18:11:31.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfcb7fe700 1 -- 172.21.15.115:0/988148668 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdfc400bd60 con 0x7fdfd40ffd00 2023-07-14T18:11:31.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd419b7e0 con 0x7fdfd40ffd00 2023-07-14T18:11:31.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfcb7fe700 1 -- 172.21.15.115:0/988148668 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4005350 con 0x7fdfd40ffd00 2023-07-14T18:11:31.400 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.356+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd419bcb0 con 0x7fdfd40ffd00 2023-07-14T18:11:31.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.357+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd4109160 con 0x7fdfd40ffd00 2023-07-14T18:11:31.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.358+0000 7fdfcb7fe700 1 -- 172.21.15.115:0/988148668 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fdfc40054b0 con 0x7fdfd40ffd00 2023-07-14T18:11:31.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.359+0000 7fdfcb7fe700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfc00a33f0 0x7fdfc00a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.359+0000 7fdfcb7fe700 1 -- 172.21.15.115:0/988148668 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdfc40ce6d0 con 0x7fdfd40ffd00 2023-07-14T18:11:31.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.359+0000 7fdfda207700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfc00a33f0 0x7fdfc00a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.359+0000 7fdfda207700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfc00a33f0 0x7fdfc00a58a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40ffb60 tx=0x7fdfd0005cf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:31.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.362+0000 7fdfcb7fe700 1 -- 172.21.15.115:0/988148668 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdfc4097a40 con 0x7fdfd40ffd00 2023-07-14T18:11:31.590 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.551+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdfd4105a90 con 0x7fdfd40ffd00 2023-07-14T18:11:31.591 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.551+0000 7fdfcb7fe700 1 -- 172.21.15.115:0/988148668 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdfc409b060 con 0x7fdfd40ffd00 2023-07-14T18:11:31.591 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:11:31.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfc00a33f0 msgr2=0x7fdfc00a58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfc00a33f0 0x7fdfc00a58a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40ffb60 tx=0x7fdfd0005cf0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 msgr2=0x7fdfd4193c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 0x7fdfd4193c50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4003a50 tx=0x7fdfc4003b30 comp rx=0 tx=0).stop 2023-07-14T18:11:31.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 shutdown_connections 2023-07-14T18:11:31.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfc00a33f0 0x7fdfc00a58a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfd4100690 0x7fdfd419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfd40feb10 0x7fdfd4193710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 --2- 172.21.15.115:0/988148668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfd40ffd00 0x7fdfd4193c50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 >> 172.21.15.115:0/988148668 conn(0x7fdfd40fa130 msgr2=0x7fdfd40fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:31.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 shutdown_connections 2023-07-14T18:11:31.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.553+0000 7fdfdc46b700 1 -- 172.21.15.115:0/988148668 wait complete. 2023-07-14T18:11:31.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:31 smithi115 ceph-mon[127451]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T18:11:31.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:31 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1638229723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:31.604 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:31 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2338725320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:31.605 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:11:31.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:31 smithi150 ceph-mon[128853]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T18:11:31.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:31 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1638229723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:31.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:31 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2338725320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:31.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:31 smithi187 ceph-mon[129317]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T18:11:31.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:31 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1638229723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:31.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:31 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2338725320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:31.729 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.689+0000 7f155ac6d700 1 -- 172.21.15.115:0/2974985665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15540ffc10 msgr2=0x7f1554100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.689+0000 7f155ac6d700 1 --2- 172.21.15.115:0/2974985665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15540ffc10 0x7f1554100020 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1548009990 tx=0x7f1548009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.689+0000 7f155ac6d700 1 -- 172.21.15.115:0/2974985665 shutdown_connections 2023-07-14T18:11:31.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.689+0000 7f155ac6d700 1 --2- 172.21.15.115:0/2974985665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1554101790 0x7f1554108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.689+0000 7f155ac6d700 1 --2- 172.21.15.115:0/2974985665 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15540ffc10 0x7f1554100020 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.689+0000 7f155ac6d700 1 --2- 172.21.15.115:0/2974985665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1554100e00 0x7f1554101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.689+0000 7f155ac6d700 1 -- 172.21.15.115:0/2974985665 >> 172.21.15.115:0/2974985665 conn(0x7f15540fb250 msgr2=0x7f15540fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:31.730 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.689+0000 7f155ac6d700 1 -- 172.21.15.115:0/2974985665 shutdown_connections 2023-07-14T18:11:31.731 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.689+0000 7f155ac6d700 1 -- 172.21.15.115:0/2974985665 wait complete. 2023-07-14T18:11:31.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.690+0000 7f155ac6d700 1 Processor -- start 2023-07-14T18:11:31.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.690+0000 7f155ac6d700 1 -- start start 2023-07-14T18:11:31.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.690+0000 7f155ac6d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15540ffc10 0x7f1554195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.690+0000 7f155ac6d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1554100e00 0x7f1554195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.690+0000 7f155ac6d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1554101790 0x7f155419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.690+0000 7f155ac6d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155419d300 con 0x7f1554101790 2023-07-14T18:11:31.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.690+0000 7f155ac6d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155419d440 con 0x7f15540ffc10 2023-07-14T18:11:31.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155ac6d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155419d5d0 con 0x7f1554100e00 2023-07-14T18:11:31.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155920a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1554101790 0x7f155419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155920a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1554101790 0x7f155419cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53700/0 (socket says 172.21.15.115:53700) 2023-07-14T18:11:31.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155920a700 1 -- 172.21.15.115:0/3567215632 learned_addr learned my addr 172.21.15.115:0/3567215632 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:31.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f1553fff700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1554100e00 0x7f1554195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f1558a09700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15540ffc10 0x7f1554195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.734 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155920a700 1 -- 172.21.15.115:0/3567215632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1554100e00 msgr2=0x7f1554195cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155920a700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1554100e00 0x7f1554195cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155920a700 1 -- 172.21.15.115:0/3567215632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15540ffc10 msgr2=0x7f1554195780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155920a700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15540ffc10 0x7f1554195780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155920a700 1 -- 172.21.15.115:0/3567215632 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480095f0 con 0x7f1554101790 2023-07-14T18:11:31.735 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f1558a09700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15540ffc10 0x7f1554195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:31.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155920a700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1554101790 0x7f155419cd30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f154c007d70 tx=0x7f154c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:31.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f1551ffb700 1 -- 172.21.15.115:0/3567215632 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c00ad40 con 0x7f1554101790 2023-07-14T18:11:31.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155419d8b0 con 0x7f1554101790 2023-07-14T18:11:31.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f1551ffb700 1 -- 172.21.15.115:0/3567215632 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f154c0128b0 con 0x7f1554101790 2023-07-14T18:11:31.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f1551ffb700 1 -- 172.21.15.115:0/3567215632 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c0115c0 con 0x7f1554101790 2023-07-14T18:11:31.736 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.691+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1554072a50 con 0x7f1554101790 2023-07-14T18:11:31.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.692+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1554106af0 con 0x7f1554101790 2023-07-14T18:11:31.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.693+0000 7f1551ffb700 1 -- 172.21.15.115:0/3567215632 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f154c012a20 con 0x7f1554101790 2023-07-14T18:11:31.737 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.694+0000 7f1551ffb700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f15440a55f0 0x7f15440a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:31.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.694+0000 7f1558a09700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f15440a55f0 0x7f15440a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:31.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.694+0000 7f1551ffb700 1 -- 172.21.15.115:0/3567215632 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f154c0ccf20 con 0x7f1554101790 2023-07-14T18:11:31.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.694+0000 7f1558a09700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f15440a55f0 0x7f15440a7aa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f154800c010 tx=0x7f1548005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:31.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.698+0000 7f1551ffb700 1 -- 172.21.15.115:0/3567215632 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f154c0962b0 con 0x7f1554101790 2023-07-14T18:11:31.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.889+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f155402cb30 con 0x7f1554101790 2023-07-14T18:11:31.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.890+0000 7f1551ffb700 1 -- 172.21.15.115:0/3567215632 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f154c0160a0 con 0x7f1554101790 2023-07-14T18:11:31.930 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 64 2023-07-14T18:11:31.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f15440a55f0 msgr2=0x7f15440a7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f15440a55f0 0x7f15440a7aa0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f154800c010 tx=0x7f1548005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1554101790 msgr2=0x7f155419cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:31.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1554101790 0x7f155419cd30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f154c007d70 tx=0x7f154c00a340 comp rx=0 tx=0).stop 2023-07-14T18:11:31.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 shutdown_connections 2023-07-14T18:11:31.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f15440a55f0 0x7f15440a7aa0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1554101790 0x7f155419cd30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15540ffc10 0x7f1554195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 --2- 172.21.15.115:0/3567215632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1554100e00 0x7f1554195cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:31.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 >> 172.21.15.115:0/3567215632 conn(0x7f15540fb250 msgr2=0x7f15540fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:31.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 shutdown_connections 2023-07-14T18:11:31.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:31.892+0000 7f155ac6d700 1 -- 172.21.15.115:0/3567215632 wait complete. 2023-07-14T18:11:31.944 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:11:32.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.028+0000 7f9df73d0700 1 -- 172.21.15.115:0/2479471700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df01000b0 msgr2=0x7f9df01004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.028+0000 7f9df73d0700 1 --2- 172.21.15.115:0/2479471700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df01000b0 0x7f9df01004c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009a10 tx=0x7f9dec009d20 comp rx=0 tx=0).stop 2023-07-14T18:11:32.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.028+0000 7f9df73d0700 1 -- 172.21.15.115:0/2479471700 shutdown_connections 2023-07-14T18:11:32.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.028+0000 7f9df73d0700 1 --2- 172.21.15.115:0/2479471700 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df00ff6b0 0x7f9df0106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.028+0000 7f9df73d0700 1 --2- 172.21.15.115:0/2479471700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df01000b0 0x7f9df01004c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.028+0000 7f9df73d0700 1 --2- 172.21.15.115:0/2479471700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df00fed00 0x7f9df00ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.028+0000 7f9df73d0700 1 -- 172.21.15.115:0/2479471700 >> 172.21.15.115:0/2479471700 conn(0x7f9df00f98a0 msgr2=0x7f9df00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:32.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.028+0000 7f9df73d0700 1 -- 172.21.15.115:0/2479471700 shutdown_connections 2023-07-14T18:11:32.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.028+0000 7f9df73d0700 1 -- 172.21.15.115:0/2479471700 wait complete. 2023-07-14T18:11:32.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df73d0700 1 Processor -- start 2023-07-14T18:11:32.071 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df73d0700 1 -- start start 2023-07-14T18:11:32.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df73d0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df00fed00 0x7f9df0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df73d0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df00ff6b0 0x7f9df0197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df73d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df01000b0 0x7f9df019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df73d0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df019f5c0 con 0x7f9df00fed00 2023-07-14T18:11:32.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df73d0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df019f700 con 0x7f9df00ff6b0 2023-07-14T18:11:32.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df73d0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df019f890 con 0x7f9df01000b0 2023-07-14T18:11:32.072 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df516c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df00fed00 0x7f9df0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df516c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df00fed00 0x7f9df0197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53710/0 (socket says 172.21.15.115:53710) 2023-07-14T18:11:32.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df516c700 1 -- 172.21.15.115:0/468870488 learned_addr learned my addr 172.21.15.115:0/468870488 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:32.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.029+0000 7f9df496b700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df00ff6b0 0x7f9df0197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9df516c700 1 -- 172.21.15.115:0/468870488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df01000b0 msgr2=0x7f9df019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9df516c700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df01000b0 0x7f9df019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9df516c700 1 -- 172.21.15.115:0/468870488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df00ff6b0 msgr2=0x7f9df0197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9df516c700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df00ff6b0 0x7f9df0197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.073 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9df516c700 1 -- 172.21.15.115:0/468870488 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec0095f0 con 0x7f9df00fed00 2023-07-14T18:11:32.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9df516c700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df00fed00 0x7f9df0197a40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9dec006010 tx=0x7f9dec00b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:32.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9de67fc700 1 -- 172.21.15.115:0/468870488 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec005040 con 0x7f9df00fed00 2023-07-14T18:11:32.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9de67fc700 1 -- 172.21.15.115:0/468870488 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9dec005630 con 0x7f9df00fed00 2023-07-14T18:11:32.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df019fb10 con 0x7f9df00fed00 2023-07-14T18:11:32.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9de67fc700 1 -- 172.21.15.115:0/468870488 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec004ea0 con 0x7f9df00fed00 2023-07-14T18:11:32.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.030+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df019ff30 con 0x7f9df00fed00 2023-07-14T18:11:32.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.031+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df01090f0 con 0x7f9df00fed00 2023-07-14T18:11:32.074 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.032+0000 7f9de67fc700 1 -- 172.21.15.115:0/468870488 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f9dec004670 con 0x7f9df00fed00 2023-07-14T18:11:32.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.032+0000 7f9de67fc700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11e0 0x7f9ddc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.032+0000 7f9de67fc700 1 -- 172.21.15.115:0/468870488 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9dec0cef00 con 0x7f9df00fed00 2023-07-14T18:11:32.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.033+0000 7f9df496b700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11e0 0x7f9ddc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.075 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.033+0000 7f9df496b700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11e0 0x7f9ddc0a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9de000bbd0 tx=0x7f9de000b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:32.076 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.037+0000 7f9de67fc700 1 -- 172.21.15.115:0/468870488 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9dec098380 con 0x7f9df00fed00 2023-07-14T18:11:32.263 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.224+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9df01a0540 con 0x7f9df00fed00 2023-07-14T18:11:32.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.224+0000 7f9de67fc700 1 -- 172.21.15.115:0/468870488 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9dec09b9a0 con 0x7f9df00fed00 2023-07-14T18:11:32.264 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:11:32.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11e0 msgr2=0x7f9ddc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11e0 0x7f9ddc0a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9de000bbd0 tx=0x7f9de000b5b0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df00fed00 msgr2=0x7f9df0197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df00fed00 0x7f9df0197a40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9dec006010 tx=0x7f9dec00b750 comp rx=0 tx=0).stop 2023-07-14T18:11:32.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 shutdown_connections 2023-07-14T18:11:32.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9ddc0a11e0 0x7f9ddc0a3690 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9df01000b0 0x7f9df019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df00fed00 0x7f9df0197a40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 --2- 172.21.15.115:0/468870488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df00ff6b0 0x7f9df0197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 >> 172.21.15.115:0/468870488 conn(0x7f9df00f98a0 msgr2=0x7f9df00fbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:32.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 shutdown_connections 2023-07-14T18:11:32.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.226+0000 7f9df73d0700 1 -- 172.21.15.115:0/468870488 wait complete. 2023-07-14T18:11:32.278 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:11:32.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.360+0000 7f186b76c700 1 -- 172.21.15.115:0/2848387446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1864100680 msgr2=0x7f1864106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.360+0000 7f186b76c700 1 --2- 172.21.15.115:0/2848387446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1864100680 0x7f1864106f40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f185c00b0a0 tx=0x7f185c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 -- 172.21.15.115:0/2848387446 shutdown_connections 2023-07-14T18:11:32.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 --2- 172.21.15.115:0/2848387446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1864100680 0x7f1864106f40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 --2- 172.21.15.115:0/2848387446 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18640feb00 0x7f18640fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 --2- 172.21.15.115:0/2848387446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18640ffcf0 0x7f1864100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 -- 172.21.15.115:0/2848387446 >> 172.21.15.115:0/2848387446 conn(0x7f18640fa100 msgr2=0x7f18640fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:32.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 -- 172.21.15.115:0/2848387446 shutdown_connections 2023-07-14T18:11:32.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 -- 172.21.15.115:0/2848387446 wait complete. 2023-07-14T18:11:32.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 Processor -- start 2023-07-14T18:11:32.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 -- start start 2023-07-14T18:11:32.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18640feb00 0x7f1864102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18640ffcf0 0x7f18641010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864100680 0x7f1864101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.404 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1864101db0 con 0x7f18640feb00 2023-07-14T18:11:32.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1864101f20 con 0x7f1864100680 2023-07-14T18:11:32.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f186b76c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186419f970 con 0x7f18640ffcf0 2023-07-14T18:11:32.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.361+0000 7f1869508700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18640feb00 0x7f1864102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869508700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18640feb00 0x7f1864102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53724/0 (socket says 172.21.15.115:53724) 2023-07-14T18:11:32.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869508700 1 -- 172.21.15.115:0/1770567473 learned_addr learned my addr 172.21.15.115:0/1770567473 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:32.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1868d07700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18640ffcf0 0x7f18641010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869d09700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864100680 0x7f1864101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869508700 1 -- 172.21.15.115:0/1770567473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18640ffcf0 msgr2=0x7f18641010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869508700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18640ffcf0 0x7f18641010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869508700 1 -- 172.21.15.115:0/1770567473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864100680 msgr2=0x7f1864101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869508700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864100680 0x7f1864101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869508700 1 -- 172.21.15.115:0/1770567473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c009d00 con 0x7f18640feb00 2023-07-14T18:11:32.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869d09700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864100680 0x7f1864101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:32.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f1869508700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18640feb00 0x7f1864102a90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f186000d980 tx=0x7f186000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:32.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f185a7fc700 1 -- 172.21.15.115:0/1770567473 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186000b950 con 0x7f18640feb00 2023-07-14T18:11:32.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f185a7fc700 1 -- 172.21.15.115:0/1770567473 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1860007620 con 0x7f18640feb00 2023-07-14T18:11:32.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186419fc50 con 0x7f18640feb00 2023-07-14T18:11:32.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f185a7fc700 1 -- 172.21.15.115:0/1770567473 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186000f3c0 con 0x7f18640feb00 2023-07-14T18:11:32.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.362+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18641a0160 con 0x7f18640feb00 2023-07-14T18:11:32.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.363+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186410a190 con 0x7f18640feb00 2023-07-14T18:11:32.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.364+0000 7f185a7fc700 1 -- 172.21.15.115:0/1770567473 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f186000bab0 con 0x7f18640feb00 2023-07-14T18:11:32.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.365+0000 7f185a7fc700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f18500a1220 0x7f18500a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.365+0000 7f1868d07700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f18500a1220 0x7f18500a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.365+0000 7f185a7fc700 1 -- 172.21.15.115:0/1770567473 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f18600cd4e0 con 0x7f18640feb00 2023-07-14T18:11:32.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.365+0000 7f1868d07700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f18500a1220 0x7f18500a36d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1854005950 tx=0x7f185400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:32.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.368+0000 7f185a7fc700 1 -- 172.21.15.115:0/1770567473 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f18600967c0 con 0x7f18640feb00 2023-07-14T18:11:32.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:32 smithi115 ceph-mon[127451]: pgmap v290: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-14T18:11:32.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:32 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/392429390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:32.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:32 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/988148668' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:32.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:32 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3567215632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:32.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:32 smithi150 ceph-mon[128853]: pgmap v290: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-14T18:11:32.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:32 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/392429390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:32.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:32 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/988148668' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:32.623 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:32 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3567215632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:32.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.600+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f186410a380 con 0x7f18640feb00 2023-07-14T18:11:32.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.600+0000 7f185a7fc700 1 -- 172.21.15.115:0/1770567473 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10851 (secure 0 0 0) 0x7f186000e350 con 0x7f18640feb00 2023-07-14T18:11:32.640 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:32.641 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:10:38.422616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24493,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24446":{"gid":24446,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:11:32.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f18500a1220 msgr2=0x7f18500a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f18500a1220 0x7f18500a36d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1854005950 tx=0x7f185400b410 comp rx=0 tx=0).stop 2023-07-14T18:11:32.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18640feb00 msgr2=0x7f1864102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18640feb00 0x7f1864102a90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f186000d980 tx=0x7f186000dc90 comp rx=0 tx=0).stop 2023-07-14T18:11:32.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 shutdown_connections 2023-07-14T18:11:32.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f18500a1220 0x7f18500a36d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864100680 0x7f1864101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18640feb00 0x7f1864102a90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 --2- 172.21.15.115:0/1770567473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f18640ffcf0 0x7f18641010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.602+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 >> 172.21.15.115:0/1770567473 conn(0x7f18640fa100 msgr2=0x7f1864067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:32.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.603+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 shutdown_connections 2023-07-14T18:11:32.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.603+0000 7f186b76c700 1 -- 172.21.15.115:0/1770567473 wait complete. 2023-07-14T18:11:32.644 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 24 2023-07-14T18:11:32.655 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:11:32.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:32 smithi187 ceph-mon[129317]: pgmap v290: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-14T18:11:32.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:32 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/392429390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:32.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:32 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/988148668' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:32.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:32 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3567215632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:32.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 -- 172.21.15.115:0/912462242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 msgr2=0x7f829c102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/912462242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 0x7f829c102ae0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8298009990 tx=0x7f8298009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 -- 172.21.15.115:0/912462242 shutdown_connections 2023-07-14T18:11:32.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/912462242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f829c105940 0x7f829c107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/912462242 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 0x7f829c102ae0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/912462242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f829c103020 0x7f829c105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 -- 172.21.15.115:0/912462242 >> 172.21.15.115:0/912462242 conn(0x7f829c0fa160 msgr2=0x7f829c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:32.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 -- 172.21.15.115:0/912462242 shutdown_connections 2023-07-14T18:11:32.777 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 -- 172.21.15.115:0/912462242 wait complete. 2023-07-14T18:11:32.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 Processor -- start 2023-07-14T18:11:32.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 -- start start 2023-07-14T18:11:32.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 0x7f829c195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f829c103020 0x7f829c195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.779 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f829c105940 0x7f829c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.736+0000 7f82a3c3d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c19d3f0 con 0x7f829c100700 2023-07-14T18:11:32.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a3c3d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c19d530 con 0x7f829c105940 2023-07-14T18:11:32.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a3c3d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c19d6c0 con 0x7f829c103020 2023-07-14T18:11:32.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a19d9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 0x7f829c195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a19d9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 0x7f829c195870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53736/0 (socket says 172.21.15.115:53736) 2023-07-14T18:11:32.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a19d9700 1 -- 172.21.15.115:0/1768625556 learned_addr learned my addr 172.21.15.115:0/1768625556 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:32.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a11d8700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f829c103020 0x7f829c195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a19d9700 1 -- 172.21.15.115:0/1768625556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f829c103020 msgr2=0x7f829c195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a19d9700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f829c103020 0x7f829c195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a19d9700 1 -- 172.21.15.115:0/1768625556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f829c105940 msgr2=0x7f829c19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a19d9700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f829c105940 0x7f829c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a19d9700 1 -- 172.21.15.115:0/1768625556 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82980095f0 con 0x7f829c100700 2023-07-14T18:11:32.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a19d9700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 0x7f829c195870 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8298000c00 tx=0x7f8298003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:32.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f8292ffd700 1 -- 172.21.15.115:0/1768625556 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829801d070 con 0x7f829c100700 2023-07-14T18:11:32.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f8292ffd700 1 -- 172.21.15.115:0/1768625556 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f829800bd70 con 0x7f829c100700 2023-07-14T18:11:32.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c19d940 con 0x7f829c100700 2023-07-14T18:11:32.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f8292ffd700 1 -- 172.21.15.115:0/1768625556 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298022cf0 con 0x7f829c100700 2023-07-14T18:11:32.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.737+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c19dd60 con 0x7f829c100700 2023-07-14T18:11:32.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.738+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829c0fbcd0 con 0x7f829c100700 2023-07-14T18:11:32.783 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.739+0000 7f8292ffd700 1 -- 172.21.15.115:0/1768625556 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f8298022470 con 0x7f829c100700 2023-07-14T18:11:32.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.740+0000 7f8292ffd700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82880a1220 0x7f82880a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:32.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.740+0000 7f82a11d8700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82880a1220 0x7f82880a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:32.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.740+0000 7f8292ffd700 1 -- 172.21.15.115:0/1768625556 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f82980ce430 con 0x7f829c100700 2023-07-14T18:11:32.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.740+0000 7f82a11d8700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82880a1220 0x7f82880a36d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f828c005ea0 tx=0x7f828c005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:32.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.744+0000 7f8292ffd700 1 -- 172.21.15.115:0/1768625556 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f82980977a0 con 0x7f829c100700 2023-07-14T18:11:32.976 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.936+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f829c02c8e0 con 0x7f829c100700 2023-07-14T18:11:32.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.937+0000 7f8292ffd700 1 -- 172.21.15.115:0/1768625556 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24558 (secure 0 0 0) 0x7f829809adc0 con 0x7f829c100700 2023-07-14T18:11:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:32.978 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":89,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:11:30.172466+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:11:32.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82880a1220 msgr2=0x7f82880a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82880a1220 0x7f82880a36d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f828c005ea0 tx=0x7f828c005e30 comp rx=0 tx=0).stop 2023-07-14T18:11:32.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 msgr2=0x7f829c195870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:32.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 0x7f829c195870 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8298000c00 tx=0x7f8298003b00 comp rx=0 tx=0).stop 2023-07-14T18:11:32.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 shutdown_connections 2023-07-14T18:11:32.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f82880a1220 0x7f82880a36d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f829c105940 0x7f829c19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f829c100700 0x7f829c195870 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 --2- 172.21.15.115:0/1768625556 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f829c103020 0x7f829c195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:32.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 >> 172.21.15.115:0/1768625556 conn(0x7f829c0fa160 msgr2=0x7f829c101360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:32.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 shutdown_connections 2023-07-14T18:11:32.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:32.940+0000 7f82a3c3d700 1 -- 172.21.15.115:0/1768625556 wait complete. 2023-07-14T18:11:32.992 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:11:33.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.073+0000 7f4871195700 1 -- 172.21.15.115:0/1553580828 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0feb00 msgr2=0x7f486c0fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.073+0000 7f4871195700 1 --2- 172.21.15.115:0/1553580828 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0feb00 0x7f486c0fef10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f486400b0a0 tx=0x7f486400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.073+0000 7f4871195700 1 -- 172.21.15.115:0/1553580828 shutdown_connections 2023-07-14T18:11:33.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.073+0000 7f4871195700 1 --2- 172.21.15.115:0/1553580828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c100680 0x7f486c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.073+0000 7f4871195700 1 --2- 172.21.15.115:0/1553580828 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0feb00 0x7f486c0fef10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.073+0000 7f4871195700 1 --2- 172.21.15.115:0/1553580828 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f486c0ffcf0 0x7f486c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.073+0000 7f4871195700 1 -- 172.21.15.115:0/1553580828 >> 172.21.15.115:0/1553580828 conn(0x7f486c0fa100 msgr2=0x7f486c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.073+0000 7f4871195700 1 -- 172.21.15.115:0/1553580828 shutdown_connections 2023-07-14T18:11:33.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.073+0000 7f4871195700 1 -- 172.21.15.115:0/1553580828 wait complete. 2023-07-14T18:11:33.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f4871195700 1 Processor -- start 2023-07-14T18:11:33.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f4871195700 1 -- start start 2023-07-14T18:11:33.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f4871195700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f486c0feb00 0x7f486c194910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f4871195700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0ffcf0 0x7f486c194e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f4871195700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c100680 0x7f486c19bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f4871195700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c19c490 con 0x7f486c0ffcf0 2023-07-14T18:11:33.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f4871195700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c19c5d0 con 0x7f486c100680 2023-07-14T18:11:33.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f4871195700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c19c760 con 0x7f486c0feb00 2023-07-14T18:11:33.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f486a59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0ffcf0 0x7f486c194e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f486a59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0ffcf0 0x7f486c194e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53752/0 (socket says 172.21.15.115:53752) 2023-07-14T18:11:33.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.074+0000 7f486a59c700 1 -- 172.21.15.115:0/1641828473 learned_addr learned my addr 172.21.15.115:0/1641828473 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f486ad9d700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f486c0feb00 0x7f486c194910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f486b59e700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c100680 0x7f486c19bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f486a59c700 1 -- 172.21.15.115:0/1641828473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f486c0feb00 msgr2=0x7f486c194910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f486a59c700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f486c0feb00 0x7f486c194910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f486a59c700 1 -- 172.21.15.115:0/1641828473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c100680 msgr2=0x7f486c19bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f486a59c700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c100680 0x7f486c19bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f486a59c700 1 -- 172.21.15.115:0/1641828473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4864009d00 con 0x7f486c0ffcf0 2023-07-14T18:11:33.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f486a59c700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0ffcf0 0x7f486c194e50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f485400c920 tx=0x7f485400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f4863fff700 1 -- 172.21.15.115:0/1641828473 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4854007b00 con 0x7f486c0ffcf0 2023-07-14T18:11:33.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486c19ca40 con 0x7f486c0ffcf0 2023-07-14T18:11:33.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f4863fff700 1 -- 172.21.15.115:0/1641828473 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f485400ce70 con 0x7f486c0ffcf0 2023-07-14T18:11:33.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f4863fff700 1 -- 172.21.15.115:0/1641828473 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48540053e0 con 0x7f486c0ffcf0 2023-07-14T18:11:33.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.075+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486c19cf50 con 0x7f486c0ffcf0 2023-07-14T18:11:33.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.076+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486c106110 con 0x7f486c0ffcf0 2023-07-14T18:11:33.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.077+0000 7f4863fff700 1 -- 172.21.15.115:0/1641828473 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f4854005560 con 0x7f486c0ffcf0 2023-07-14T18:11:33.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.078+0000 7f4863fff700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f48580a1220 0x7f48580a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.078+0000 7f486ad9d700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f48580a1220 0x7f48580a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.078+0000 7f4863fff700 1 -- 172.21.15.115:0/1641828473 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f48540ccdb0 con 0x7f486c0ffcf0 2023-07-14T18:11:33.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.078+0000 7f486ad9d700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f48580a1220 0x7f48580a36d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4864009fd0 tx=0x7f486400b7f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.081+0000 7f4863fff700 1 -- 172.21.15.115:0/1641828473 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4854096090 con 0x7f486c0ffcf0 2023-07-14T18:11:33.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.313+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f486c02c8e0 con 0x7f486c0ffcf0 2023-07-14T18:11:33.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.314+0000 7f4863fff700 1 -- 172.21.15.115:0/1641828473 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10851 (secure 0 0 0) 0x7f485401d020 con 0x7f486c0ffcf0 2023-07-14T18:11:33.354 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:33.354 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:10:38.422616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24493,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24446":{"gid":24446,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:11:33.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f48580a1220 msgr2=0x7f48580a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f48580a1220 0x7f48580a36d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4864009fd0 tx=0x7f486400b7f0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0ffcf0 msgr2=0x7f486c194e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0ffcf0 0x7f486c194e50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f485400c920 tx=0x7f485400cc30 comp rx=0 tx=0).stop 2023-07-14T18:11:33.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 shutdown_connections 2023-07-14T18:11:33.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f48580a1220 0x7f48580a36d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c100680 0x7f486c19bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f486c0feb00 0x7f486c194910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 --2- 172.21.15.115:0/1641828473 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f486c0ffcf0 0x7f486c194e50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 >> 172.21.15.115:0/1641828473 conn(0x7f486c0fa100 msgr2=0x7f486c103250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 shutdown_connections 2023-07-14T18:11:33.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.316+0000 7f4871195700 1 -- 172.21.15.115:0/1641828473 wait complete. 2023-07-14T18:11:33.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:33 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/468870488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:33.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:33 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1770567473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:33.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:33 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1768625556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:33.359 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 24 2023-07-14T18:11:33.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T18:10:17.634244+0000', 'modified': '2023-07-14T18:10:38.422616+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24515, 'mds_1': 24493, 'mds_2': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24446': {'gid': 24446, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6835/3630781789', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 3630781789}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 3630781789}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.187:6835/4258537735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 4258537735}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 4258537735}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.187:6837/771796341', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6836', 'nonce': 771796341}, {'type': 'v1', 'addr': '172.21.15.187:6837', 'nonce': 771796341}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6837/1462100932', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6836', 'nonce': 1462100932}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 1462100932}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.187:6839/1449243194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6838', 'nonce': 1449243194}, {'type': 'v1', 'addr': '172.21.15.187:6839', 'nonce': 1449243194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6839/2895627668', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 2895627668}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 2895627668}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T18:11:33.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-14T18:11:33.369 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-07-14T18:11:33.492 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 -- 172.21.15.115:0/1638421524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 msgr2=0x7f0824101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 --2- 172.21.15.115:0/1638421524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 0x7f0824101280 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0814009990 tx=0x7f0814009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 -- 172.21.15.115:0/1638421524 shutdown_connections 2023-07-14T18:11:33.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 --2- 172.21.15.115:0/1638421524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08241017c0 0x7f0824108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 --2- 172.21.15.115:0/1638421524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08240ffc40 0x7f0824100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 --2- 172.21.15.115:0/1638421524 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 0x7f0824101280 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.493 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 -- 172.21.15.115:0/1638421524 >> 172.21.15.115:0/1638421524 conn(0x7f08240fb240 msgr2=0x7f08240fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 -- 172.21.15.115:0/1638421524 shutdown_connections 2023-07-14T18:11:33.494 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 -- 172.21.15.115:0/1638421524 wait complete. 2023-07-14T18:11:33.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 Processor -- start 2023-07-14T18:11:33.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 -- start start 2023-07-14T18:11:33.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08240ffc40 0x7f08241958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 0x7f0824195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08241017c0 0x7f082419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082419d470 con 0x7f0824100e30 2023-07-14T18:11:33.496 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082419d5b0 con 0x7f08240ffc40 2023-07-14T18:11:33.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.453+0000 7f082b65f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082419d740 con 0x7f08241017c0 2023-07-14T18:11:33.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0828bfa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 0x7f0824195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0828bfa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 0x7f0824195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53764/0 (socket says 172.21.15.115:53764) 2023-07-14T18:11:33.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0828bfa700 1 -- 172.21.15.115:0/475965745 learned_addr learned my addr 172.21.15.115:0/475965745 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0829bfc700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08241017c0 0x7f082419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f08293fb700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08240ffc40 0x7f08241958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0828bfa700 1 -- 172.21.15.115:0/475965745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08241017c0 msgr2=0x7f082419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0828bfa700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08241017c0 0x7f082419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0828bfa700 1 -- 172.21.15.115:0/475965745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08240ffc40 msgr2=0x7f08241958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0828bfa700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08240ffc40 0x7f08241958f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0828bfa700 1 -- 172.21.15.115:0/475965745 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140095f0 con 0x7f0824100e30 2023-07-14T18:11:33.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0829bfc700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08241017c0 0x7f082419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:33.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f0828bfa700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 0x7f0824195e30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0814006010 tx=0x7f0814005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f081a7fc700 1 -- 172.21.15.115:0/475965745 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081401d070 con 0x7f0824100e30 2023-07-14T18:11:33.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f081a7fc700 1 -- 172.21.15.115:0/475965745 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0814004100 con 0x7f0824100e30 2023-07-14T18:11:33.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f082b65f700 1 -- 172.21.15.115:0/475965745 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082419d9c0 con 0x7f0824100e30 2023-07-14T18:11:33.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f081a7fc700 1 -- 172.21.15.115:0/475965745 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0814005290 con 0x7f0824100e30 2023-07-14T18:11:33.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.454+0000 7f082b65f700 1 -- 172.21.15.115:0/475965745 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082419de40 con 0x7f0824100e30 2023-07-14T18:11:33.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.456+0000 7f081a7fc700 1 -- 172.21.15.115:0/475965745 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f08140047e0 con 0x7f0824100e30 2023-07-14T18:11:33.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.457+0000 7f081a7fc700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f08100a1260 0x7f08100a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.457+0000 7f081a7fc700 1 -- 172.21.15.115:0/475965745 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f08140cddf0 con 0x7f0824100e30 2023-07-14T18:11:33.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.457+0000 7f08293fb700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f08100a1260 0x7f08100a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.504 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.457+0000 7f08293fb700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f08100a1260 0x7f08100a3710 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0820005950 tx=0x7f08200058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f08377f7b80 1 Processor -- start 2023-07-14T18:11:33.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f08377f7b80 1 -- start start 2023-07-14T18:11:33.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa0f10 0x55b496aa1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496aa1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 0x55b496aa28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f08377f7b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4969a5c60 con 0x55b496aa1860 2023-07-14T18:11:33.505 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f08377f7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4969f89f0 con 0x55b496aa2460 2023-07-14T18:11:33.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f08377f7b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4969f1e80 con 0x55b496aa0f10 2023-07-14T18:11:33.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f0828bfa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496aa1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f0828bfa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496aa1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53774/0 (socket says 172.21.15.115:53774) 2023-07-14T18:11:33.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f0828bfa700 1 -- 172.21.15.115:0/1287768440 learned_addr learned my addr 172.21.15.115:0/1287768440 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f08293fb700 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa0f10 0x55b496aa1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.458+0000 7f0829bfc700 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 0x55b496aa28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f0828bfa700 1 -- 172.21.15.115:0/1287768440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa0f10 msgr2=0x55b496aa1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.506 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f0828bfa700 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa0f10 0x55b496aa1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f0828bfa700 1 -- 172.21.15.115:0/1287768440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 msgr2=0x55b496aa28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f0828bfa700 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 0x55b496aa28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f0828bfa700 1 -- 172.21.15.115:0/1287768440 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b49681fd90 con 0x55b496aa1860 2023-07-14T18:11:33.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f0829bfc700 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 0x55b496aa28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:33.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f0828bfa700 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496aa1e80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0814097890 tx=0x7f081409aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1ded5ef121acbc server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f080f7fe700 1 -- 172.21.15.115:0/1287768440 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081409ee50 con 0x55b496aa1860 2023-07-14T18:11:33.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f080f7fe700 1 -- 172.21.15.115:0/1287768440 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f08140a31e0 con 0x55b496aa1860 2023-07-14T18:11:33.507 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.459+0000 7f080f7fe700 1 -- 172.21.15.115:0/1287768440 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08140a2fb0 con 0x55b496aa1860 2023-07-14T18:11:33.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 -- 172.21.15.115:0/1287768440 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 msgr2=0x55b496aa1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496aa1e80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0814097890 tx=0x7f081409aa70 comp rx=0 tx=0).stop 2023-07-14T18:11:33.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 -- 172.21.15.115:0/1287768440 shutdown_connections 2023-07-14T18:11:33.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 0x55b496aa28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa0f10 0x55b496aa1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 --2- 172.21.15.115:0/1287768440 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496aa1e80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 -- 172.21.15.115:0/1287768440 >> 172.21.15.115:0/1287768440 conn(0x55b496a96630 msgr2=0x55b496a9b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 -- 172.21.15.115:0/1287768440 shutdown_connections 2023-07-14T18:11:33.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 -- 172.21.15.115:0/1287768440 wait complete. 2023-07-14T18:11:33.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 -- start start 2023-07-14T18:11:33.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.460+0000 7f08377f7b80 1 Processor -- start 2023-07-14T18:11:33.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 0x55b496aaa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 0x55b496aaab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa2460 0x55b496ab1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.509 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f08377f7b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b496991a70 con 0x55b496aa2460 2023-07-14T18:11:33.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f08377f7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b496991770 con 0x55b496aa0f10 2023-07-14T18:11:33.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f08377f7b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4968758d0 con 0x55b496aa1860 2023-07-14T18:11:33.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f08377f7b80 10 client.0 _ll_register_callbacks cb 0x55b49698f710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:33.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0829bfc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa2460 0x55b496ab1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0829bfc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa2460 0x55b496ab1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53782/0 (socket says 172.21.15.115:53782) 2023-07-14T18:11:33.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0829bfc700 1 -- 172.21.15.115:0/565196206 learned_addr learned my addr 172.21.15.115:0/565196206 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f08293fb700 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 0x55b496aaa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.510 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0828bfa700 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 0x55b496aaab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0829bfc700 1 -- 172.21.15.115:0/565196206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 msgr2=0x55b496aaab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0829bfc700 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 0x55b496aaab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0829bfc700 1 -- 172.21.15.115:0/565196206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 msgr2=0x55b496aaa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.511 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0829bfc700 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 0x55b496aaa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0829bfc700 1 -- 172.21.15.115:0/565196206 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082419de40 con 0x55b496aa2460 2023-07-14T18:11:33.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f0829bfc700 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa2460 0x55b496ab1b40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f081c007b70 tx=0x7f081c007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77008d822e947189 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f080cff9700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:33.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f080cff9700 1 -- 172.21.15.115:0/565196206 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c018070 con 0x55b496aa2460 2023-07-14T18:11:33.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f080cff9700 1 -- 172.21.15.115:0/565196206 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f081c003c10 con 0x55b496aa2460 2023-07-14T18:11:33.512 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.461+0000 7f08377f7b80 1 -- 172.21.15.115:0/565196206 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b49694b4a0 con 0x55b496aa2460 2023-07-14T18:11:33.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f080cff9700 1 -- 172.21.15.115:0/565196206 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f081c003dd0 con 0x55b496aa2460 2023-07-14T18:11:33.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f08377f7b80 10 client.15219 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:33.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f08377f7b80 1 -- 172.21.15.115:0/565196206 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b496808c10 con 0x55b496aa2460 2023-07-14T18:11:33.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f080cff9700 1 -- 172.21.15.115:0/565196206 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55b49694b4a0 con 0x55b496aa2460 2023-07-14T18:11:33.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f080cff9700 1 -- 172.21.15.115:0/565196206 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f081c01b020 con 0x55b496aa2460 2023-07-14T18:11:33.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f08377f7b80 10 client.15219 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:33.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f08377f7b80 10 client.15219 resolve_mds: resolved name 'f' to daemon mds.f 2023-07-14T18:11:33.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f08377f7b80 20 client.15219 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:33.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f08377f7b80 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55b496ab5830 0x55b496ab4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f08377f7b80 4 client.15219 mds_command: new command op to 24463 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:11:33.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f08377f7b80 1 -- 172.21.15.115:0/565196206 --> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b496a6fd50 con 0x55b496ab5830 2023-07-14T18:11:33.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.462+0000 7f08293fb700 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55b496ab5830 0x55b496ab4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.514 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.463+0000 7f080cff9700 10 client.15219 ms_handle_connect on v2:172.21.15.187:6834/4258537735 2023-07-14T18:11:33.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.463+0000 7f08293fb700 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55b496ab5830 0x55b496ab4c10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d183e4c90e366480 server_cookie=ae93fad9b0268fa2 in_seq=0 out_seq=0 2023-07-14T18:11:33.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.463+0000 7f080cff9700 1 -- 172.21.15.115:0/565196206 <== mds.2 v2:172.21.15.187:6834/4258537735 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55b496a6fd50 con 0x55b496ab5830 2023-07-14T18:11:33.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.463+0000 7f080cff9700 10 client.15219 handle_command_reply: tid=0 2023-07-14T18:11:33.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.463+0000 7f08377f7b80 1 client.15219 shutdown 2023-07-14T18:11:33.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 -- 172.21.15.115:0/565196206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa2460 msgr2=0x55b496ab1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa2460 0x55b496ab1b40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f081c007b70 tx=0x7f081c007e80 comp rx=0 tx=0).stop 2023-07-14T18:11:33.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 -- 172.21.15.115:0/565196206 shutdown_connections 2023-07-14T18:11:33.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55b496ab5830 0x55b496ab4c10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa2460 0x55b496ab1b40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 0x55b496aaa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f080cff9700 0 client.15219 ms_handle_reset on v2:172.21.15.187:6834/4258537735 2023-07-14T18:11:33.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 --2- 172.21.15.115:0/565196206 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 0x55b496aaab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 -- 172.21.15.115:0/565196206 >> 172.21.15.115:0/565196206 conn(0x55b496a96630 msgr2=0x55b496a9a600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 -- 172.21.15.115:0/565196206 shutdown_connections 2023-07-14T18:11:33.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 1 -- 172.21.15.115:0/565196206 wait complete. 2023-07-14T18:11:33.518 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.464+0000 7f08377f7b80 20 client.15219 trim_cache size 0 max 16384 2023-07-14T18:11:33.522 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08377f7b80 1 Processor -- start 2023-07-14T18:11:33.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08377f7b80 1 -- start start 2023-07-14T18:11:33.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa0f10 0x55b496a9b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 0x55b496ab1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 0x55b496aa8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.523 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08377f7b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b496a6fd50 con 0x55b496aa0f10 2023-07-14T18:11:33.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08377f7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b49694b4a0 con 0x55b496aa2460 2023-07-14T18:11:33.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08377f7b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b496a986b0 con 0x55b496aa1860 2023-07-14T18:11:33.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08293fb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa0f10 0x55b496a9b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08293fb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa0f10 0x55b496a9b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53786/0 (socket says 172.21.15.115:53786) 2023-07-14T18:11:33.524 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08293fb700 1 -- 172.21.15.115:0/1250769748 learned_addr learned my addr 172.21.15.115:0/1250769748 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f0828bfa700 1 --2- 172.21.15.115:0/1250769748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 0x55b496ab1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f0829bfc700 1 --2- 172.21.15.115:0/1250769748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 0x55b496aa8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08293fb700 1 -- 172.21.15.115:0/1250769748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 msgr2=0x55b496ab1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08293fb700 1 --2- 172.21.15.115:0/1250769748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 0x55b496ab1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08293fb700 1 -- 172.21.15.115:0/1250769748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 msgr2=0x55b496aa8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08293fb700 1 --2- 172.21.15.115:0/1250769748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 0x55b496aa8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08293fb700 1 -- 172.21.15.115:0/1250769748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b496846200 con 0x55b496aa0f10 2023-07-14T18:11:33.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.481+0000 7f08293fb700 1 --2- 172.21.15.115:0/1250769748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa0f10 0x55b496a9b0a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0820009470 tx=0x7f0820014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92963f4341d035eb server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f080ffff700 1 -- 172.21.15.115:0/1250769748 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0820025040 con 0x55b496aa0f10 2023-07-14T18:11:33.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f080ffff700 1 -- 172.21.15.115:0/1250769748 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0820011d20 con 0x55b496aa0f10 2023-07-14T18:11:33.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f080ffff700 1 -- 172.21.15.115:0/1250769748 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082001f650 con 0x55b496aa0f10 2023-07-14T18:11:33.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f08377f7b80 1 -- 172.21.15.115:0/1250769748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa0f10 msgr2=0x55b496a9b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f08377f7b80 1 --2- 172.21.15.115:0/1250769748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa0f10 0x55b496a9b0a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0820009470 tx=0x7f0820014000 comp rx=0 tx=0).stop 2023-07-14T18:11:33.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f08377f7b80 1 -- 172.21.15.115:0/1250769748 shutdown_connections 2023-07-14T18:11:33.526 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f08377f7b80 1 --2- 172.21.15.115:0/1250769748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa2460 0x55b496aa8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f08377f7b80 1 --2- 172.21.15.115:0/1250769748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa0f10 0x55b496a9b0a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f08377f7b80 1 --2- 172.21.15.115:0/1250769748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa1860 0x55b496ab1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f08377f7b80 1 -- 172.21.15.115:0/1250769748 >> 172.21.15.115:0/1250769748 conn(0x55b496a96630 msgr2=0x55b496a97b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f08377f7b80 1 -- 172.21.15.115:0/1250769748 shutdown_connections 2023-07-14T18:11:33.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.482+0000 7f08377f7b80 1 -- 172.21.15.115:0/1250769748 wait complete. 2023-07-14T18:11:33.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08377f7b80 1 -- start start 2023-07-14T18:11:33.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08377f7b80 1 Processor -- start 2023-07-14T18:11:33.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 0x55b496aafa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496b07480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08377f7b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa2460 0x55b496b099d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08377f7b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b496aa80c0 con 0x55b496aa1860 2023-07-14T18:11:33.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08377f7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b496b09f10 con 0x55b496aa0f10 2023-07-14T18:11:33.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08377f7b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b496b0a0a0 con 0x55b496aa2460 2023-07-14T18:11:33.531 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08377f7b80 10 client.0 _ll_register_callbacks cb 0x7f081c009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:33.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f0828bfa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496b07480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f0828bfa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496b07480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53794/0 (socket says 172.21.15.115:53794) 2023-07-14T18:11:33.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f0828bfa700 1 -- 172.21.15.115:0/349018814 learned_addr learned my addr 172.21.15.115:0/349018814 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f08293fb700 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 0x55b496aafa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f0828bfa700 1 -- 172.21.15.115:0/349018814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa2460 msgr2=0x55b496b099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.483+0000 7f0828bfa700 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa2460 0x55b496b099d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f0828bfa700 1 -- 172.21.15.115:0/349018814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 msgr2=0x55b496aafa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f0828bfa700 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 0x55b496aafa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f0828bfa700 1 -- 172.21.15.115:0/349018814 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082000e970 con 0x55b496aa1860 2023-07-14T18:11:33.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08293fb700 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 0x55b496aafa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:33.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f0828bfa700 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496b07480 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f081400c010 tx=0x7f08140cb910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a412f3ab202da5eb server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f080cff9700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:33.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f080cff9700 1 -- 172.21.15.115:0/349018814 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08140991c0 con 0x55b496aa1860 2023-07-14T18:11:33.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f080cff9700 1 -- 172.21.15.115:0/349018814 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f08140a2f60 con 0x55b496aa1860 2023-07-14T18:11:33.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08377f7b80 1 -- 172.21.15.115:0/349018814 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f08140a31e0 con 0x55b496aa1860 2023-07-14T18:11:33.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f080cff9700 1 -- 172.21.15.115:0/349018814 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0814003c70 con 0x55b496aa1860 2023-07-14T18:11:33.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08377f7b80 10 client.15225 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:33.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08377f7b80 1 -- 172.21.15.115:0/349018814 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b49681e260 con 0x55b496aa1860 2023-07-14T18:11:33.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f080cff9700 1 -- 172.21.15.115:0/349018814 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f08140cb020 con 0x55b496aa1860 2023-07-14T18:11:33.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f080cff9700 1 -- 172.21.15.115:0/349018814 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f08140053f0 con 0x55b496aa1860 2023-07-14T18:11:33.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08377f7b80 10 client.15225 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:33.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08377f7b80 10 client.15225 resolve_mds: resolved name 'f' to daemon mds.f 2023-07-14T18:11:33.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08377f7b80 20 client.15225 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:33.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08377f7b80 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55b496ab5830 0x55b496b0b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08377f7b80 4 client.15225 mds_command: new command op to 24463 tid=0[{"prefix": "status"}] 2023-07-14T18:11:33.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.484+0000 7f08377f7b80 1 -- 172.21.15.115:0/349018814 --> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b4969eae60 con 0x55b496ab5830 2023-07-14T18:11:33.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08293fb700 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55b496ab5830 0x55b496b0b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f080cff9700 10 client.15225 ms_handle_connect on v2:172.21.15.187:6834/4258537735 2023-07-14T18:11:33.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08293fb700 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55b496ab5830 0x55b496b0b2f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fe635f5e628ac3e4 server_cookie=103c7737a27c2ed9 in_seq=0 out_seq=0 2023-07-14T18:11:33.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f080cff9700 1 -- 172.21.15.115:0/349018814 <== mds.2 v2:172.21.15.187:6834/4258537735 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55b4969eae60 con 0x55b496ab5830 2023-07-14T18:11:33.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f080cff9700 10 client.15225 handle_command_reply: tid=0 2023-07-14T18:11:33.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 client.15225 shutdown 2023-07-14T18:11:33.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 -- 172.21.15.115:0/349018814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 msgr2=0x55b496b07480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496b07480 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f081400c010 tx=0x7f08140cb910 comp rx=0 tx=0).stop 2023-07-14T18:11:33.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 -- 172.21.15.115:0/349018814 shutdown_connections 2023-07-14T18:11:33.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55b496aa2460 0x55b496b099d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b496aa0f10 0x55b496aafa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55b496ab5830 0x55b496b0b2f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 --2- 172.21.15.115:0/349018814 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b496aa1860 0x55b496b07480 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f080cff9700 0 client.15225 ms_handle_reset on v2:172.21.15.187:6834/4258537735 2023-07-14T18:11:33.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 -- 172.21.15.115:0/349018814 >> 172.21.15.115:0/349018814 conn(0x55b496a96630 msgr2=0x55b496a9b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 -- 172.21.15.115:0/349018814 shutdown_connections 2023-07-14T18:11:33.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.485+0000 7f08377f7b80 1 -- 172.21.15.115:0/349018814 wait complete. 2023-07-14T18:11:33.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f08377f7b80 20 client.15225 trim_cache size 0 max 16384 2023-07-14T18:11:33.538 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:11:33.538 INFO:teuthology.orchestra.run.smithi115.stdout: "cluster_fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:11:33.538 INFO:teuthology.orchestra.run.smithi115.stdout: "whoami": 2, 2023-07-14T18:11:33.538 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 24463, 2023-07-14T18:11:33.538 INFO:teuthology.orchestra.run.smithi115.stdout: "want_state": "up:active", 2023-07-14T18:11:33.538 INFO:teuthology.orchestra.run.smithi115.stdout: "state": "up:active", 2023-07-14T18:11:33.539 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs", 2023-07-14T18:11:33.539 INFO:teuthology.orchestra.run.smithi115.stdout: "rank_uptime": 56.067000169000003, 2023-07-14T18:11:33.539 INFO:teuthology.orchestra.run.smithi115.stdout: "mdsmap_epoch": 24, 2023-07-14T18:11:33.539 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch": 89, 2023-07-14T18:11:33.539 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch_barrier": 87, 2023-07-14T18:11:33.539 INFO:teuthology.orchestra.run.smithi115.stdout: "uptime": 95.113918952999995 2023-07-14T18:11:33.539 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:11:33.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 -- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f08100a1260 msgr2=0x7f08100a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f08100a1260 0x7f08100a3710 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0820005950 tx=0x7f08200058e0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 -- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 msgr2=0x7f0824195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 0x7f0824195e30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0814006010 tx=0x7f0814005ef0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 -- 172.21.15.115:0/475965745 shutdown_connections 2023-07-14T18:11:33.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f08100a1260 0x7f08100a3710 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f08241017c0 0x7f082419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08240ffc40 0x7f08241958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 --2- 172.21.15.115:0/475965745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0824100e30 0x7f0824195e30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 -- 172.21.15.115:0/475965745 >> 172.21.15.115:0/475965745 conn(0x7f08240fb240 msgr2=0x7f08240fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 -- 172.21.15.115:0/475965745 shutdown_connections 2023-07-14T18:11:33.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.486+0000 7f082b65f700 1 -- 172.21.15.115:0/475965745 wait complete. 2023-07-14T18:11:33.544 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T18:11:33.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:33 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/468870488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:33.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:33 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1770567473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:33.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:33 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1768625556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:33.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 -- 172.21.15.115:0/3528292011 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 msgr2=0x7fc150107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.669 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 --2- 172.21.15.115:0/3528292011 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 0x7fc150107010 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc14800b0a0 tx=0x7fc14800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 -- 172.21.15.115:0/3528292011 shutdown_connections 2023-07-14T18:11:33.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 --2- 172.21.15.115:0/3528292011 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 0x7fc150107010 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 --2- 172.21.15.115:0/3528292011 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500feb30 0x7fc1500fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 --2- 172.21.15.115:0/3528292011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1500ffdc0 0x7fc150100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 -- 172.21.15.115:0/3528292011 >> 172.21.15.115:0/3528292011 conn(0x7fc1500fa150 msgr2=0x7fc1500fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 -- 172.21.15.115:0/3528292011 shutdown_connections 2023-07-14T18:11:33.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 -- 172.21.15.115:0/3528292011 wait complete. 2023-07-14T18:11:33.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 Processor -- start 2023-07-14T18:11:33.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.628+0000 7fc157243700 1 -- start start 2023-07-14T18:11:33.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc157243700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500feb30 0x7fc1501959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc157243700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1500ffdc0 0x7fc150195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc157243700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 0x7fc15019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc157243700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15019d550 con 0x7fc150100770 2023-07-14T18:11:33.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc157243700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15019d690 con 0x7fc1500ffdc0 2023-07-14T18:11:33.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc157243700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15019d820 con 0x7fc1500feb30 2023-07-14T18:11:33.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc1557e0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 0x7fc15019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc1557e0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 0x7fc15019cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53810/0 (socket says 172.21.15.115:53810) 2023-07-14T18:11:33.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc1557e0700 1 -- 172.21.15.115:0/1032036542 learned_addr learned my addr 172.21.15.115:0/1032036542 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc154fdf700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500feb30 0x7fc1501959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc14ffff700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1500ffdc0 0x7fc150195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc1557e0700 1 -- 172.21.15.115:0/1032036542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500feb30 msgr2=0x7fc1501959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc1557e0700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500feb30 0x7fc1501959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc1557e0700 1 -- 172.21.15.115:0/1032036542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1500ffdc0 msgr2=0x7fc150195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc1557e0700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1500ffdc0 0x7fc150195f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc1557e0700 1 -- 172.21.15.115:0/1032036542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009d00 con 0x7fc150100770 2023-07-14T18:11:33.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc154fdf700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500feb30 0x7fc1501959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:33.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc1557e0700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 0x7fc15019cf80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc14800b780 tx=0x7fc14800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc14dffb700 1 -- 172.21.15.115:0/1032036542 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14800e040 con 0x7fc150100770 2023-07-14T18:11:33.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc14dffb700 1 -- 172.21.15.115:0/1032036542 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc148004e60 con 0x7fc150100770 2023-07-14T18:11:33.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc157243700 1 -- 172.21.15.115:0/1032036542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15019daa0 con 0x7fc150100770 2023-07-14T18:11:33.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc14dffb700 1 -- 172.21.15.115:0/1032036542 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1480052a0 con 0x7fc150100770 2023-07-14T18:11:33.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.629+0000 7fc157243700 1 -- 172.21.15.115:0/1032036542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15019df20 con 0x7fc150100770 2023-07-14T18:11:33.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.632+0000 7fc14dffb700 1 -- 172.21.15.115:0/1032036542 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fc14801d990 con 0x7fc150100770 2023-07-14T18:11:33.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.632+0000 7fc14dffb700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc1400a11e0 0x7fc1400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.632+0000 7fc14dffb700 1 -- 172.21.15.115:0/1032036542 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc1480ce4f0 con 0x7fc150100770 2023-07-14T18:11:33.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.632+0000 7fc154fdf700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc1400a11e0 0x7fc1400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc154fdf700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc1400a11e0 0x7fc1400a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc144009cb0 tx=0x7fc144009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1633dbb80 1 Processor -- start 2023-07-14T18:11:33.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1633dbb80 1 -- start start 2023-07-14T18:11:33.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 0x56199ae0b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 0x56199ae0be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae0c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ad0fc60 con 0x56199ae0c460 2023-07-14T18:11:33.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ad629f0 con 0x56199ae0af10 2023-07-14T18:11:33.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ad5be80 con 0x56199ae0b860 2023-07-14T18:11:33.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1557e0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae0c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1557e0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae0c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53818/0 (socket says 172.21.15.115:53818) 2023-07-14T18:11:33.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1557e0700 1 -- 172.21.15.115:0/1301208238 learned_addr learned my addr 172.21.15.115:0/1301208238 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc154fdf700 1 --2- 172.21.15.115:0/1301208238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 0x56199ae0b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc14ffff700 1 --2- 172.21.15.115:0/1301208238 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 0x56199ae0be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1557e0700 1 -- 172.21.15.115:0/1301208238 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 msgr2=0x56199ae0be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1557e0700 1 --2- 172.21.15.115:0/1301208238 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 0x56199ae0be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1557e0700 1 -- 172.21.15.115:0/1301208238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 msgr2=0x56199ae0b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1557e0700 1 --2- 172.21.15.115:0/1301208238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 0x56199ae0b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.633+0000 7fc1557e0700 1 -- 172.21.15.115:0/1301208238 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56199ab89d90 con 0x56199ae0c460 2023-07-14T18:11:33.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1557e0700 1 --2- 172.21.15.115:0/1301208238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae0c8d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc148097f90 tx=0x7fc14809b370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9a816892e4408d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc13affd700 1 -- 172.21.15.115:0/1301208238 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14809b520 con 0x56199ae0c460 2023-07-14T18:11:33.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc13affd700 1 -- 172.21.15.115:0/1301208238 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc14809f550 con 0x56199ae0c460 2023-07-14T18:11:33.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc13affd700 1 -- 172.21.15.115:0/1301208238 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14809f120 con 0x56199ae0c460 2023-07-14T18:11:33.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1633dbb80 1 -- 172.21.15.115:0/1301208238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 msgr2=0x56199ae0c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/1301208238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae0c8d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc148097f90 tx=0x7fc14809b370 comp rx=0 tx=0).stop 2023-07-14T18:11:33.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1633dbb80 1 -- 172.21.15.115:0/1301208238 shutdown_connections 2023-07-14T18:11:33.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/1301208238 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae0c8d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/1301208238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 0x56199ae0b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/1301208238 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 0x56199ae0be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1633dbb80 1 -- 172.21.15.115:0/1301208238 >> 172.21.15.115:0/1301208238 conn(0x56199ae00630 msgr2=0x56199ae05980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1633dbb80 1 -- 172.21.15.115:0/1301208238 shutdown_connections 2023-07-14T18:11:33.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.634+0000 7fc1633dbb80 1 -- 172.21.15.115:0/1301208238 wait complete. 2023-07-14T18:11:33.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.635+0000 7fc1633dbb80 1 -- start start 2023-07-14T18:11:33.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.635+0000 7fc1633dbb80 1 Processor -- start 2023-07-14T18:11:33.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.635+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 0x56199ae14690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.635+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0b860 0x56199ae14bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.635+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae1bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.635+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199acfba70 con 0x56199ae0c460 2023-07-14T18:11:33.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.635+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199acfb770 con 0x56199ae0b860 2023-07-14T18:11:33.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.635+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199abdf8d0 con 0x56199ae0af10 2023-07-14T18:11:33.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1633dbb80 10 client.0 _ll_register_callbacks cb 0x56199acf9710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:33.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1557e0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae1bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1557e0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae1bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53824/0 (socket says 172.21.15.115:53824) 2023-07-14T18:11:33.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1557e0700 1 -- 172.21.15.115:0/726594589 learned_addr learned my addr 172.21.15.115:0/726594589 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc14ffff700 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0b860 0x56199ae14bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc154fdf700 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 0x56199ae14690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1557e0700 1 -- 172.21.15.115:0/726594589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 msgr2=0x56199ae14690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1557e0700 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 0x56199ae14690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1557e0700 1 -- 172.21.15.115:0/726594589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0b860 msgr2=0x56199ae14bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.687 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1557e0700 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0b860 0x56199ae14bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1557e0700 1 -- 172.21.15.115:0/726594589 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15019df20 con 0x56199ae0c460 2023-07-14T18:11:33.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc14ffff700 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0b860 0x56199ae14bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:33.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1557e0700 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae1bc10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc148026000 tx=0x7fc14809af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378954f3d42c26cc server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc12ffff700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:33.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc12ffff700 1 -- 172.21.15.115:0/726594589 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14809f120 con 0x56199ae0c460 2023-07-14T18:11:33.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc12ffff700 1 -- 172.21.15.115:0/726594589 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc1480b0020 con 0x56199ae0c460 2023-07-14T18:11:33.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1633dbb80 1 -- 172.21.15.115:0/726594589 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56199acb54a0 con 0x56199ae0c460 2023-07-14T18:11:33.688 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc12ffff700 1 -- 172.21.15.115:0/726594589 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc14809f280 con 0x56199ae0c460 2023-07-14T18:11:33.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1633dbb80 10 client.15237 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:33.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc1633dbb80 1 -- 172.21.15.115:0/726594589 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56199ab72c10 con 0x56199ae0c460 2023-07-14T18:11:33.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.636+0000 7fc12ffff700 1 -- 172.21.15.115:0/726594589 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x56199acb54a0 con 0x56199ae0c460 2023-07-14T18:11:33.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc12ffff700 1 -- 172.21.15.115:0/726594589 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc14809aaa0 con 0x56199ae0c460 2023-07-14T18:11:33.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc1633dbb80 10 client.15237 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:33.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc1633dbb80 10 client.15237 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T18:11:33.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc1633dbb80 20 client.15237 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:33.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x56199ae1f750 0x56199ae23e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc1633dbb80 4 client.15237 mds_command: new command op to 24493 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:11:33.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc1633dbb80 1 -- 172.21.15.115:0/726594589 --> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56199add9d50 con 0x56199ae1f750 2023-07-14T18:11:33.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc154fdf700 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x56199ae1f750 0x56199ae23e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc12ffff700 10 client.15237 ms_handle_connect on v2:172.21.15.187:6838/1449243194 2023-07-14T18:11:33.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.637+0000 7fc154fdf700 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x56199ae1f750 0x56199ae23e90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=68ce257fc8cbdcf server_cookie=3ad3410b1b5a6f77 in_seq=0 out_seq=0 2023-07-14T18:11:33.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc12ffff700 1 -- 172.21.15.115:0/726594589 <== mds.1 v2:172.21.15.187:6838/1449243194 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x56199add9d50 con 0x56199ae1f750 2023-07-14T18:11:33.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc12ffff700 10 client.15237 handle_command_reply: tid=0 2023-07-14T18:11:33.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 client.15237 shutdown 2023-07-14T18:11:33.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 -- 172.21.15.115:0/726594589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 msgr2=0x56199ae1bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae1bc10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc148026000 tx=0x7fc14809af40 comp rx=0 tx=0).stop 2023-07-14T18:11:33.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 -- 172.21.15.115:0/726594589 shutdown_connections 2023-07-14T18:11:33.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x56199ae1f750 0x56199ae23e90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae1bc10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 0x56199ae14690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/726594589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0b860 0x56199ae14bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc12ffff700 0 client.15237 ms_handle_reset on v2:172.21.15.187:6838/1449243194 2023-07-14T18:11:33.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 -- 172.21.15.115:0/726594589 >> 172.21.15.115:0/726594589 conn(0x56199ae00630 msgr2=0x56199ae04600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 -- 172.21.15.115:0/726594589 shutdown_connections 2023-07-14T18:11:33.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 1 -- 172.21.15.115:0/726594589 wait complete. 2023-07-14T18:11:33.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.638+0000 7fc1633dbb80 20 client.15237 trim_cache size 0 max 16384 2023-07-14T18:11:33.693 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:33 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/468870488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:33.693 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:33 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1770567473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:33.693 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:33 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1768625556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:33.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc1633dbb80 1 Processor -- start 2023-07-14T18:11:33.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc1633dbb80 1 -- start start 2023-07-14T18:11:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 0x56199ae01b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0b860 0x56199ae050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0c460 0x56199ae127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ad05470 con 0x56199ae0b860 2023-07-14T18:11:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ad2a790 con 0x56199ae0c460 2023-07-14T18:11:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199abebde0 con 0x56199ae0af10 2023-07-14T18:11:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc14ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0b860 0x56199ae050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc14ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0b860 0x56199ae050a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53830/0 (socket says 172.21.15.115:53830) 2023-07-14T18:11:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc14ffff700 1 -- 172.21.15.115:0/2084045968 learned_addr learned my addr 172.21.15.115:0/2084045968 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc1557e0700 1 --2- 172.21.15.115:0/2084045968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0c460 0x56199ae127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc154fdf700 1 --2- 172.21.15.115:0/2084045968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 0x56199ae01b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc14ffff700 1 -- 172.21.15.115:0/2084045968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 msgr2=0x56199ae01b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc14ffff700 1 --2- 172.21.15.115:0/2084045968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 0x56199ae01b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc14ffff700 1 -- 172.21.15.115:0/2084045968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0c460 msgr2=0x56199ae127d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc14ffff700 1 --2- 172.21.15.115:0/2084045968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0c460 0x56199ae127d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.656+0000 7fc14ffff700 1 -- 172.21.15.115:0/2084045968 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56199abb0200 con 0x56199ae0b860 2023-07-14T18:11:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc14ffff700 1 --2- 172.21.15.115:0/2084045968 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0b860 0x56199ae050a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00fba0 tx=0x7fc13c00feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966d94d00ed5d9f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc13b7fe700 1 -- 172.21.15.115:0/2084045968 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c011b30 con 0x56199ae0b860 2023-07-14T18:11:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc13b7fe700 1 -- 172.21.15.115:0/2084045968 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc13c016070 con 0x56199ae0b860 2023-07-14T18:11:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc13b7fe700 1 -- 172.21.15.115:0/2084045968 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c012450 con 0x56199ae0b860 2023-07-14T18:11:33.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc1633dbb80 1 -- 172.21.15.115:0/2084045968 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0b860 msgr2=0x56199ae050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/2084045968 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0b860 0x56199ae050a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00fba0 tx=0x7fc13c00feb0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc1633dbb80 1 -- 172.21.15.115:0/2084045968 shutdown_connections 2023-07-14T18:11:33.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/2084045968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0c460 0x56199ae127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/2084045968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0af10 0x56199ae01b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/2084045968 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0b860 0x56199ae050a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc1633dbb80 1 -- 172.21.15.115:0/2084045968 >> 172.21.15.115:0/2084045968 conn(0x56199ae00630 msgr2=0x56199ae05950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc1633dbb80 1 -- 172.21.15.115:0/2084045968 shutdown_connections 2023-07-14T18:11:33.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.657+0000 7fc1633dbb80 1 -- 172.21.15.115:0/2084045968 wait complete. 2023-07-14T18:11:33.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1633dbb80 1 -- start start 2023-07-14T18:11:33.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1633dbb80 1 Processor -- start 2023-07-14T18:11:33.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 0x56199ae052f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 0x56199ae05830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1633dbb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae77320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ae77860 con 0x56199ae0c460 2023-07-14T18:11:33.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ae779a0 con 0x56199ae0af10 2023-07-14T18:11:33.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1633dbb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ae77ae0 con 0x56199ae0b860 2023-07-14T18:11:33.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1633dbb80 10 client.0 _ll_register_callbacks cb 0x7fc144009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:33.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1557e0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae77320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1557e0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae77320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53838/0 (socket says 172.21.15.115:53838) 2023-07-14T18:11:33.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc1557e0700 1 -- 172.21.15.115:0/3883168767 learned_addr learned my addr 172.21.15.115:0/3883168767 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc154fdf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 0x56199ae052f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.658+0000 7fc14ffff700 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 0x56199ae05830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc154fdf700 1 -- 172.21.15.115:0/3883168767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 msgr2=0x56199ae05830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc154fdf700 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 0x56199ae05830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc154fdf700 1 -- 172.21.15.115:0/3883168767 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 msgr2=0x56199ae77320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc154fdf700 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae77320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc154fdf700 1 -- 172.21.15.115:0/3883168767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c00f820 con 0x56199ae0af10 2023-07-14T18:11:33.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc14ffff700 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 0x56199ae05830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:11:33.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc154fdf700 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 0x56199ae052f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x56199ae20410 tx=0x7fc144014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ede2169674ed5126 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc12ffff700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-07-14T18:11:33.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc12ffff700 1 -- 172.21.15.115:0/3883168767 <== mon.2 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144028af0 con 0x56199ae0af10 2023-07-14T18:11:33.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc12ffff700 1 -- 172.21.15.115:0/3883168767 <== mon.2 v2:172.21.15.150:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc144018030 con 0x56199ae0af10 2023-07-14T18:11:33.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc1633dbb80 1 -- 172.21.15.115:0/3883168767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc14809f550 con 0x56199ae0af10 2023-07-14T18:11:33.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc12ffff700 1 -- 172.21.15.115:0/3883168767 <== mon.2 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1440284a0 con 0x56199ae0af10 2023-07-14T18:11:33.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc1633dbb80 10 client.24923 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:33.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc1633dbb80 1 -- 172.21.15.115:0/3883168767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56199ab88260 con 0x56199ae0af10 2023-07-14T18:11:33.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.659+0000 7fc12ffff700 1 -- 172.21.15.115:0/3883168767 <== mon.2 v2:172.21.15.150:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc14809f550 con 0x56199ae0af10 2023-07-14T18:11:33.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc12ffff700 1 -- 172.21.15.115:0/3883168767 <== mon.2 v2:172.21.15.150:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc14402f030 con 0x56199ae0af10 2023-07-14T18:11:33.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc1633dbb80 10 client.24923 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:33.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc1633dbb80 10 client.24923 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T18:11:33.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc1633dbb80 20 client.24923 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:33.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x56199ae7c820 0x56199ae79130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc1633dbb80 4 client.24923 mds_command: new command op to 24493 tid=0[{"prefix": "status"}] 2023-07-14T18:11:33.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc1633dbb80 1 -- 172.21.15.115:0/3883168767 --> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56199ad54e60 con 0x56199ae7c820 2023-07-14T18:11:33.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc14ffff700 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x56199ae7c820 0x56199ae79130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc12ffff700 10 client.24923 ms_handle_connect on v2:172.21.15.187:6838/1449243194 2023-07-14T18:11:33.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.660+0000 7fc14ffff700 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x56199ae7c820 0x56199ae79130 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6f45651f2ccfd257 server_cookie=e928e3d1d269d05b in_seq=0 out_seq=0 2023-07-14T18:11:33.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc12ffff700 1 -- 172.21.15.115:0/3883168767 <== mds.1 v2:172.21.15.187:6838/1449243194 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x56199ad54e60 con 0x56199ae7c820 2023-07-14T18:11:33.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc12ffff700 10 client.24923 handle_command_reply: tid=0 2023-07-14T18:11:33.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 client.24923 shutdown 2023-07-14T18:11:33.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 -- 172.21.15.115:0/3883168767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 msgr2=0x56199ae052f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 0x56199ae052f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x56199ae20410 tx=0x7fc144014000 comp rx=0 tx=0).stop 2023-07-14T18:11:33.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 -- 172.21.15.115:0/3883168767 shutdown_connections 2023-07-14T18:11:33.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x56199ae7c820 0x56199ae79130 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x56199ae0c460 0x56199ae77320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56199ae0af10 0x56199ae052f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 --2- 172.21.15.115:0/3883168767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x56199ae0b860 0x56199ae05830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc12ffff700 0 client.24923 ms_handle_reset on v2:172.21.15.187:6838/1449243194 2023-07-14T18:11:33.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 -- 172.21.15.115:0/3883168767 >> 172.21.15.115:0/3883168767 conn(0x56199ae00630 msgr2=0x56199ae127d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 -- 172.21.15.115:0/3883168767 shutdown_connections 2023-07-14T18:11:33.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 1 -- 172.21.15.115:0/3883168767 wait complete. 2023-07-14T18:11:33.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc1633dbb80 20 client.24923 trim_cache size 0 max 16384 2023-07-14T18:11:33.711 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:11:33.711 INFO:teuthology.orchestra.run.smithi115.stdout: "cluster_fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:11:33.711 INFO:teuthology.orchestra.run.smithi115.stdout: "whoami": 1, 2023-07-14T18:11:33.711 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 24493, 2023-07-14T18:11:33.711 INFO:teuthology.orchestra.run.smithi115.stdout: "want_state": "up:active", 2023-07-14T18:11:33.711 INFO:teuthology.orchestra.run.smithi115.stdout: "state": "up:active", 2023-07-14T18:11:33.712 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs", 2023-07-14T18:11:33.712 INFO:teuthology.orchestra.run.smithi115.stdout: "rank_uptime": 56.690056439000003, 2023-07-14T18:11:33.712 INFO:teuthology.orchestra.run.smithi115.stdout: "mdsmap_epoch": 24, 2023-07-14T18:11:33.712 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch": 89, 2023-07-14T18:11:33.712 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch_barrier": 87, 2023-07-14T18:11:33.712 INFO:teuthology.orchestra.run.smithi115.stdout: "uptime": 88.882794598000004 2023-07-14T18:11:33.712 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:11:33.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 -- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc1400a11e0 msgr2=0x7fc1400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc1400a11e0 0x7fc1400a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc144009cb0 tx=0x7fc144009260 comp rx=0 tx=0).stop 2023-07-14T18:11:33.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 -- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 msgr2=0x7fc15019cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 0x7fc15019cf80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc14800b780 tx=0x7fc14800bdf0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 -- 172.21.15.115:0/1032036542 shutdown_connections 2023-07-14T18:11:33.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc1400a11e0 0x7fc1400a3690 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc150100770 0x7fc15019cf80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1500feb30 0x7fc1501959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 --2- 172.21.15.115:0/1032036542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1500ffdc0 0x7fc150195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.661+0000 7fc157243700 1 -- 172.21.15.115:0/1032036542 >> 172.21.15.115:0/1032036542 conn(0x7fc1500fa150 msgr2=0x7fc1501032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.662+0000 7fc157243700 1 -- 172.21.15.115:0/1032036542 shutdown_connections 2023-07-14T18:11:33.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.662+0000 7fc157243700 1 -- 172.21.15.115:0/1032036542 wait complete. 2023-07-14T18:11:33.715 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T18:11:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.797+0000 7fa7b6a85700 1 -- 172.21.15.115:0/4113141752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b01006c0 msgr2=0x7fa7b0102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.797+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/4113141752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b01006c0 0x7fa7b0102aa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009990 tx=0x7fa7a4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.797+0000 7fa7b6a85700 1 -- 172.21.15.115:0/4113141752 shutdown_connections 2023-07-14T18:11:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.797+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/4113141752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b0105900 0x7fa7b0107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.797+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/4113141752 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b01006c0 0x7fa7b0102aa0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.797+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/4113141752 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b0102fe0 0x7fa7b01053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.797+0000 7fa7b6a85700 1 -- 172.21.15.115:0/4113141752 >> 172.21.15.115:0/4113141752 conn(0x7fa7b00fa160 msgr2=0x7fa7b00fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.797+0000 7fa7b6a85700 1 -- 172.21.15.115:0/4113141752 shutdown_connections 2023-07-14T18:11:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.797+0000 7fa7b6a85700 1 -- 172.21.15.115:0/4113141752 wait complete. 2023-07-14T18:11:33.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b6a85700 1 Processor -- start 2023-07-14T18:11:33.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b6a85700 1 -- start start 2023-07-14T18:11:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b6a85700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b01006c0 0x7fa7b0195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b6a85700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b0102fe0 0x7fa7b0195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b6a85700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b0105900 0x7fa7b019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b6a85700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b019d390 con 0x7fa7b0102fe0 2023-07-14T18:11:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b6a85700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b019d4d0 con 0x7fa7b0105900 2023-07-14T18:11:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b6a85700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b019d660 con 0x7fa7b01006c0 2023-07-14T18:11:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7affff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b0102fe0 0x7fa7b0195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7affff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b0102fe0 0x7fa7b0195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53850/0 (socket says 172.21.15.115:53850) 2023-07-14T18:11:33.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7affff700 1 -- 172.21.15.115:0/3193281322 learned_addr learned my addr 172.21.15.115:0/3193281322 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b4821700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b01006c0 0x7fa7b0195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b5022700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b0105900 0x7fa7b019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7affff700 1 -- 172.21.15.115:0/3193281322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b01006c0 msgr2=0x7fa7b0195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7affff700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b01006c0 0x7fa7b0195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7affff700 1 -- 172.21.15.115:0/3193281322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b0105900 msgr2=0x7fa7b019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7affff700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b0105900 0x7fa7b019cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7affff700 1 -- 172.21.15.115:0/3193281322 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a40095f0 con 0x7fa7b0102fe0 2023-07-14T18:11:33.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b5022700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b0105900 0x7fa7b019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7affff700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b0102fe0 0x7fa7b0195d50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00d980 tx=0x7fa79c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7adffb700 1 -- 172.21.15.115:0/3193281322 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c004220 con 0x7fa7b0102fe0 2023-07-14T18:11:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.798+0000 7fa7b6a85700 1 -- 172.21.15.115:0/3193281322 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b019d940 con 0x7fa7b0102fe0 2023-07-14T18:11:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.799+0000 7fa7adffb700 1 -- 172.21.15.115:0/3193281322 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa79c007640 con 0x7fa7b0102fe0 2023-07-14T18:11:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.799+0000 7fa7adffb700 1 -- 172.21.15.115:0/3193281322 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c0055e0 con 0x7fa7b0102fe0 2023-07-14T18:11:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.799+0000 7fa7b6a85700 1 -- 172.21.15.115:0/3193281322 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b019dea0 con 0x7fa7b0102fe0 2023-07-14T18:11:33.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.801+0000 7fa7adffb700 1 -- 172.21.15.115:0/3193281322 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fa79c004750 con 0x7fa7b0102fe0 2023-07-14T18:11:33.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.801+0000 7fa7adffb700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa7a00a11e0 0x7fa7a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.801+0000 7fa7adffb700 1 -- 172.21.15.115:0/3193281322 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa79c0ccc40 con 0x7fa7b0102fe0 2023-07-14T18:11:33.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.801+0000 7fa7b4821700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa7a00a11e0 0x7fa7a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7b4821700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa7a00a11e0 0x7fa7a00a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400b5c0 tx=0x7fa7a4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7c2c1db80 1 Processor -- start 2023-07-14T18:11:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7c2c1db80 1 -- start start 2023-07-14T18:11:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc56cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cb55c60 con 0x564e9cc56850 2023-07-14T18:11:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cba89f0 con 0x564e9cc552d0 2023-07-14T18:11:33.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cba1e80 con 0x564e9cc55c20 2023-07-14T18:11:33.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7b5022700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc56cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7b5022700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc56cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53860/0 (socket says 172.21.15.115:53860) 2023-07-14T18:11:33.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7b5022700 1 -- 172.21.15.115:0/2847414703 learned_addr learned my addr 172.21.15.115:0/2847414703 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7b4821700 1 --2- 172.21.15.115:0/2847414703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.802+0000 7fa7affff700 1 --2- 172.21.15.115:0/2847414703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa7b5022700 1 -- 172.21.15.115:0/2847414703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 msgr2=0x564e9cc562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa7b5022700 1 --2- 172.21.15.115:0/2847414703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa7b5022700 1 -- 172.21.15.115:0/2847414703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 msgr2=0x564e9cc556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa7b5022700 1 --2- 172.21.15.115:0/2847414703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa7b5022700 1 -- 172.21.15.115:0/2847414703 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e9c9cfd90 con 0x564e9cc56850 2023-07-14T18:11:33.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa7b5022700 1 --2- 172.21.15.115:0/2847414703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc56cc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800d300 tx=0x7fa7a8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337bc20bb612f394 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa79affd700 1 -- 172.21.15.115:0/2847414703 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a80125f0 con 0x564e9cc56850 2023-07-14T18:11:33.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa79affd700 1 -- 172.21.15.115:0/2847414703 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa7a8012be0 con 0x564e9cc56850 2023-07-14T18:11:33.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa79affd700 1 -- 172.21.15.115:0/2847414703 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8011a00 con 0x564e9cc56850 2023-07-14T18:11:33.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2847414703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 msgr2=0x564e9cc56cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.803+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2847414703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc56cc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800d300 tx=0x7fa7a8007f90 comp rx=0 tx=0).stop 2023-07-14T18:11:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2847414703 shutdown_connections 2023-07-14T18:11:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2847414703 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc56cc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2847414703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2847414703 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2847414703 >> 172.21.15.115:0/2847414703 conn(0x564e9cc46630 msgr2=0x564e9cc4b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2847414703 shutdown_connections 2023-07-14T18:11:33.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2847414703 wait complete. 2023-07-14T18:11:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 -- start start 2023-07-14T18:11:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 Processor -- start 2023-07-14T18:11:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc552d0 0x564e9cc5ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc55c20 0x564e9cc5f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc66270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cb41a70 con 0x564e9cc56850 2023-07-14T18:11:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cb41770 con 0x564e9cc55c20 2023-07-14T18:11:33.853 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9ca258d0 con 0x564e9cc552d0 2023-07-14T18:11:33.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.804+0000 7fa7c2c1db80 10 client.0 _ll_register_callbacks cb 0x564e9cb3f710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:33.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b5022700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc66270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b5022700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc66270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53866/0 (socket says 172.21.15.115:53866) 2023-07-14T18:11:33.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b5022700 1 -- 172.21.15.115:0/1083867937 learned_addr learned my addr 172.21.15.115:0/1083867937 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b4821700 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc552d0 0x564e9cc5ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7affff700 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc55c20 0x564e9cc5f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.854 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b5022700 1 -- 172.21.15.115:0/1083867937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc552d0 msgr2=0x564e9cc5ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b5022700 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc552d0 0x564e9cc5ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b5022700 1 -- 172.21.15.115:0/1083867937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc55c20 msgr2=0x564e9cc5f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b5022700 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc55c20 0x564e9cc5f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b5022700 1 -- 172.21.15.115:0/1083867937 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a8007d70 con 0x564e9cc56850 2023-07-14T18:11:33.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7affff700 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc55c20 0x564e9cc5f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:33.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7b5022700 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc66270 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800d2d0 tx=0x7fa7a8004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f828064636df31e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa78ffff700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:33.855 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa78ffff700 1 -- 172.21.15.115:0/1083867937 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8012880 con 0x564e9cc56850 2023-07-14T18:11:33.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa78ffff700 1 -- 172.21.15.115:0/1083867937 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa7a80129e0 con 0x564e9cc56850 2023-07-14T18:11:33.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/1083867937 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564e9cafb4a0 con 0x564e9cc56850 2023-07-14T18:11:33.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa78ffff700 1 -- 172.21.15.115:0/1083867937 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7a8011780 con 0x564e9cc56850 2023-07-14T18:11:33.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7c2c1db80 10 client.15255 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:33.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/1083867937 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564e9c9b8c10 con 0x564e9cc56850 2023-07-14T18:11:33.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.805+0000 7fa78ffff700 1 -- 172.21.15.115:0/1083867937 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x564e9cafb4a0 con 0x564e9cc56850 2023-07-14T18:11:33.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa78ffff700 1 -- 172.21.15.115:0/1083867937 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa7a80054a0 con 0x564e9cc56850 2023-07-14T18:11:33.856 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa7c2c1db80 10 client.15255 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:33.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa7c2c1db80 10 client.15255 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:11:33.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa7c2c1db80 20 client.15255 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:33.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x564e9cc69fb0 0x564e9cc69360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa7c2c1db80 4 client.15255 mds_command: new command op to 24515 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:11:33.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/1083867937 --> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e9cc1fd50 con 0x564e9cc69fb0 2023-07-14T18:11:33.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa7b4821700 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x564e9cc69fb0 0x564e9cc69360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa78ffff700 10 client.15255 ms_handle_connect on v2:172.21.15.150:6838/2895627668 2023-07-14T18:11:33.857 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.806+0000 7fa7b4821700 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x564e9cc69fb0 0x564e9cc69360 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9740d640e663987b server_cookie=4b95af31f1ec8e95 in_seq=0 out_seq=0 2023-07-14T18:11:33.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa78ffff700 1 -- 172.21.15.115:0/1083867937 <== mds.0 v2:172.21.15.150:6838/2895627668 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x564e9cc1fd50 con 0x564e9cc69fb0 2023-07-14T18:11:33.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa78ffff700 10 client.15255 handle_command_reply: tid=0 2023-07-14T18:11:33.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 client.15255 shutdown 2023-07-14T18:11:33.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/1083867937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 msgr2=0x564e9cc66270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc66270 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800d2d0 tx=0x7fa7a8004f90 comp rx=0 tx=0).stop 2023-07-14T18:11:33.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/1083867937 shutdown_connections 2023-07-14T18:11:33.858 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x564e9cc69fb0 0x564e9cc69360 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc56850 0x564e9cc66270 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa78ffff700 0 client.15255 ms_handle_reset on v2:172.21.15.150:6838/2895627668 2023-07-14T18:11:33.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc552d0 0x564e9cc5ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/1083867937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc55c20 0x564e9cc5f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/1083867937 >> 172.21.15.115:0/1083867937 conn(0x564e9cc46630 msgr2=0x564e9cc4a600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/1083867937 shutdown_connections 2023-07-14T18:11:33.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.807+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/1083867937 wait complete. 2023-07-14T18:11:33.859 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.808+0000 7fa7c2c1db80 20 client.15255 trim_cache size 0 max 16384 2023-07-14T18:11:33.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7c2c1db80 1 Processor -- start 2023-07-14T18:11:33.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7c2c1db80 1 -- start start 2023-07-14T18:11:33.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc4b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc72d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc69360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cb707c0 con 0x564e9cc46630 2023-07-14T18:11:33.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9ca31e10 con 0x564e9cc552d0 2023-07-14T18:11:33.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cb3dd50 con 0x564e9cc55c20 2023-07-14T18:11:33.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7b4821700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc4b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7b4821700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc4b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53870/0 (socket says 172.21.15.115:53870) 2023-07-14T18:11:33.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7b4821700 1 -- 172.21.15.115:0/949386277 learned_addr learned my addr 172.21.15.115:0/949386277 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.825+0000 7fa7affff700 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc72d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa7b5022700 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc69360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.867 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa7b4821700 1 -- 172.21.15.115:0/949386277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 msgr2=0x564e9cc69360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa7b4821700 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc69360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa7b4821700 1 -- 172.21.15.115:0/949386277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 msgr2=0x564e9cc72d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa7b4821700 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc72d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa7b4821700 1 -- 172.21.15.115:0/949386277 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e9c9f6200 con 0x564e9cc46630 2023-07-14T18:11:33.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa7b5022700 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc69360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:33.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa7b4821700 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc4b0a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c010 tx=0x7fa7a4023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d702edb6a9bc93 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa79b7fe700 1 -- 172.21.15.115:0/949386277 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4005620 con 0x564e9cc46630 2023-07-14T18:11:33.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa79b7fe700 1 -- 172.21.15.115:0/949386277 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa7a402c040 con 0x564e9cc46630 2023-07-14T18:11:33.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.826+0000 7fa79b7fe700 1 -- 172.21.15.115:0/949386277 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4004760 con 0x564e9cc46630 2023-07-14T18:11:33.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/949386277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 msgr2=0x564e9cc4b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc4b0a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c010 tx=0x7fa7a4023000 comp rx=0 tx=0).stop 2023-07-14T18:11:33.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/949386277 shutdown_connections 2023-07-14T18:11:33.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc69360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc4b0a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/949386277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc72d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/949386277 >> 172.21.15.115:0/949386277 conn(0x564e9cc5dda0 msgr2=0x564e9cc47b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/949386277 shutdown_connections 2023-07-14T18:11:33.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/949386277 wait complete. 2023-07-14T18:11:33.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 -- start start 2023-07-14T18:11:33.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 Processor -- start 2023-07-14T18:11:33.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc6e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc6e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cc71be0 con 0x564e9cc46630 2023-07-14T18:11:33.943 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cc71d50 con 0x564e9cc552d0 2023-07-14T18:11:33.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e9cc71f10 con 0x564e9cc55c20 2023-07-14T18:11:33.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.827+0000 7fa7c2c1db80 10 client.0 _ll_register_callbacks cb 0x7fa7a800d150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:33.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b4821700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc6e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b4821700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc6e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53876/0 (socket says 172.21.15.115:53876) 2023-07-14T18:11:33.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b4821700 1 -- 172.21.15.115:0/2499831690 learned_addr learned my addr 172.21.15.115:0/2499831690 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:33.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b5022700 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.944 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7affff700 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc6e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b4821700 1 -- 172.21.15.115:0/2499831690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 msgr2=0x564e9cc716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b4821700 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b4821700 1 -- 172.21.15.115:0/2499831690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 msgr2=0x564e9cc6e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:33.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b4821700 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc6e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b4821700 1 -- 172.21.15.115:0/2499831690 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b0108950 con 0x564e9cc46630 2023-07-14T18:11:33.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7b4821700 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc6e170 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40098e0 tx=0x7fa7a4034b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7d6f51c23a988f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:33.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa78ffff700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:33.945 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa78ffff700 1 -- 172.21.15.115:0/2499831690 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4034050 con 0x564e9cc46630 2023-07-14T18:11:33.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa78ffff700 1 -- 172.21.15.115:0/2499831690 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa7a40341b0 con 0x564e9cc46630 2023-07-14T18:11:33.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2499831690 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7a8012be0 con 0x564e9cc46630 2023-07-14T18:11:33.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa78ffff700 1 -- 172.21.15.115:0/2499831690 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7a4034370 con 0x564e9cc46630 2023-07-14T18:11:33.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7c2c1db80 10 client.15261 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:33.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2499831690 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564e9c9ce260 con 0x564e9cc46630 2023-07-14T18:11:33.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.828+0000 7fa78ffff700 1 -- 172.21.15.115:0/2499831690 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa7a8012be0 con 0x564e9cc46630 2023-07-14T18:11:33.946 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa78ffff700 1 -- 172.21.15.115:0/2499831690 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa7a4003890 con 0x564e9cc46630 2023-07-14T18:11:33.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa7c2c1db80 10 client.15261 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:33.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa7c2c1db80 10 client.15261 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:11:33.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa7c2c1db80 20 client.15261 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:33.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x564e9cc56850 0x564e9ccbfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:33.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa7c2c1db80 4 client.15261 mds_command: new command op to 24515 tid=0[{"prefix": "status"}] 2023-07-14T18:11:33.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2499831690 --> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564e9cb9ae60 con 0x564e9cc56850 2023-07-14T18:11:33.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa7affff700 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x564e9cc56850 0x564e9ccbfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:33.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa78ffff700 10 client.15261 ms_handle_connect on v2:172.21.15.150:6838/2895627668 2023-07-14T18:11:33.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.829+0000 7fa7affff700 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x564e9cc56850 0x564e9ccbfc60 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=219213a5f4480412 server_cookie=5ae12c8a7ee3235a in_seq=0 out_seq=0 2023-07-14T18:11:33.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa78ffff700 1 -- 172.21.15.115:0/2499831690 <== mds.0 v2:172.21.15.150:6838/2895627668 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x564e9cb9ae60 con 0x564e9cc56850 2023-07-14T18:11:33.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa78ffff700 10 client.15261 handle_command_reply: tid=0 2023-07-14T18:11:33.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 client.15261 shutdown 2023-07-14T18:11:33.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2499831690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 msgr2=0x564e9cc6e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc6e170 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40098e0 tx=0x7fa7a4034b30 comp rx=0 tx=0).stop 2023-07-14T18:11:33.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2499831690 shutdown_connections 2023-07-14T18:11:33.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x564e9cc56850 0x564e9ccbfc60 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564e9cc55c20 0x564e9cc716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.949 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x564e9cc46630 0x564e9cc6e170 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa78ffff700 0 client.15261 ms_handle_reset on v2:172.21.15.150:6838/2895627668 2023-07-14T18:11:33.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 --2- 172.21.15.115:0/2499831690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e9cc552d0 0x564e9cc6e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2499831690 >> 172.21.15.115:0/2499831690 conn(0x564e9cc5dda0 msgr2=0x564e9cc4b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2499831690 shutdown_connections 2023-07-14T18:11:33.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 1 -- 172.21.15.115:0/2499831690 wait complete. 2023-07-14T18:11:33.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7c2c1db80 20 client.15261 trim_cache size 0 max 16384 2023-07-14T18:11:33.951 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:11:33.951 INFO:teuthology.orchestra.run.smithi115.stdout: "cluster_fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:11:33.951 INFO:teuthology.orchestra.run.smithi115.stdout: "whoami": 0, 2023-07-14T18:11:33.951 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 24515, 2023-07-14T18:11:33.951 INFO:teuthology.orchestra.run.smithi115.stdout: "want_state": "up:active", 2023-07-14T18:11:33.951 INFO:teuthology.orchestra.run.smithi115.stdout: "state": "up:active", 2023-07-14T18:11:33.951 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs", 2023-07-14T18:11:33.952 INFO:teuthology.orchestra.run.smithi115.stdout: "rank_uptime": 75.419874511000003, 2023-07-14T18:11:33.952 INFO:teuthology.orchestra.run.smithi115.stdout: "mdsmap_epoch": 24, 2023-07-14T18:11:33.952 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch": 89, 2023-07-14T18:11:33.952 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch_barrier": 87, 2023-07-14T18:11:33.952 INFO:teuthology.orchestra.run.smithi115.stdout: "uptime": 82.271834186000007 2023-07-14T18:11:33.952 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:11:33.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7b6a85700 1 -- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa7a00a11e0 msgr2=0x7fa7a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa7a00a11e0 0x7fa7a00a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400b5c0 tx=0x7fa7a4005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7b6a85700 1 -- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b0102fe0 msgr2=0x7fa7b0195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:33.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.830+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b0102fe0 0x7fa7b0195d50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00d980 tx=0x7fa79c00dc90 comp rx=0 tx=0).stop 2023-07-14T18:11:33.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.831+0000 7fa7b6a85700 1 -- 172.21.15.115:0/3193281322 shutdown_connections 2023-07-14T18:11:33.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.831+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa7a00a11e0 0x7fa7a00a3690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.831+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7b0105900 0x7fa7b019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.831+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa7b01006c0 0x7fa7b0195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.831+0000 7fa7b6a85700 1 --2- 172.21.15.115:0/3193281322 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa7b0102fe0 0x7fa7b0195d50 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:33.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.831+0000 7fa7b6a85700 1 -- 172.21.15.115:0/3193281322 >> 172.21.15.115:0/3193281322 conn(0x7fa7b00fa160 msgr2=0x7fa7b0101320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:33.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.831+0000 7fa7b6a85700 1 -- 172.21.15.115:0/3193281322 shutdown_connections 2023-07-14T18:11:33.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:33.831+0000 7fa7b6a85700 1 -- 172.21.15.115:0/3193281322 wait complete. 2023-07-14T18:11:33.955 INFO:tasks.cephadm:Setup complete, yielding 2023-07-14T18:11:33.956 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T18:11:33.965 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi115.front.sepia.ceph.com 2023-07-14T18:11:33.966 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- bash -c 'ceph orch status' 2023-07-14T18:11:34.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:34 smithi115 ceph-mon[127451]: pgmap v291: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 18 B/s, 0 objects/s recovering 2023-07-14T18:11:34.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:34 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1641828473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:34.515 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:34.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:34 smithi150 ceph-mon[128853]: pgmap v291: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 18 B/s, 0 objects/s recovering 2023-07-14T18:11:34.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:34 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1641828473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:34.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:34 smithi187 ceph-mon[129317]: pgmap v291: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 18 B/s, 0 objects/s recovering 2023-07-14T18:11:34.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:34 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1641828473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:35.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.797+0000 7f7037f31700 1 -- 172.21.15.115:0/2135083150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70300ffdb0 msgr2=0x7f7030100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:35.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.797+0000 7f7037f31700 1 --2- 172.21.15.115:0/2135083150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70300ffdb0 0x7f7030100200 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f702c009990 tx=0x7f702c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:35.839 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.797+0000 7f7037f31700 1 -- 172.21.15.115:0/2135083150 shutdown_connections 2023-07-14T18:11:35.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.797+0000 7f7037f31700 1 --2- 172.21.15.115:0/2135083150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7030100740 0x7f7030107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:35.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.797+0000 7f7037f31700 1 --2- 172.21.15.115:0/2135083150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70300febb0 0x7f70300fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:35.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.797+0000 7f7037f31700 1 --2- 172.21.15.115:0/2135083150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f70300ffdb0 0x7f7030100200 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:35.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.797+0000 7f7037f31700 1 -- 172.21.15.115:0/2135083150 >> 172.21.15.115:0/2135083150 conn(0x7f70300fa1e0 msgr2=0x7f70300fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:35.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 -- 172.21.15.115:0/2135083150 shutdown_connections 2023-07-14T18:11:35.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 -- 172.21.15.115:0/2135083150 wait complete. 2023-07-14T18:11:35.840 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 Processor -- start 2023-07-14T18:11:35.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 -- start start 2023-07-14T18:11:35.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70300febb0 0x7f7030195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:35.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70300ffdb0 0x7f7030195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:35.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7030100740 0x7f703019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:35.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703019d3c0 con 0x7f7030100740 2023-07-14T18:11:35.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703019d500 con 0x7f70300febb0 2023-07-14T18:11:35.841 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.798+0000 7f7037f31700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703019d690 con 0x7f70300ffdb0 2023-07-14T18:11:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70364ce700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7030100740 0x7f703019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70364ce700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7030100740 0x7f703019cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53892/0 (socket says 172.21.15.115:53892) 2023-07-14T18:11:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70364ce700 1 -- 172.21.15.115:0/2454787991 learned_addr learned my addr 172.21.15.115:0/2454787991 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70354cc700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70300ffdb0 0x7f7030195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f7035ccd700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70300febb0 0x7f7030195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:35.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70364ce700 1 -- 172.21.15.115:0/2454787991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70300ffdb0 msgr2=0x7f7030195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:35.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70364ce700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70300ffdb0 0x7f7030195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:35.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70364ce700 1 -- 172.21.15.115:0/2454787991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70300febb0 msgr2=0x7f7030195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:35.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70364ce700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70300febb0 0x7f7030195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:35.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70364ce700 1 -- 172.21.15.115:0/2454787991 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c0095f0 con 0x7f7030100740 2023-07-14T18:11:35.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f70364ce700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7030100740 0x7f703019cdf0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7028007da0 tx=0x7f702800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:35.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f7026ffd700 1 -- 172.21.15.115:0/2454787991 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702800ad00 con 0x7f7030100740 2023-07-14T18:11:35.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f7026ffd700 1 -- 172.21.15.115:0/2454787991 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f702800ae60 con 0x7f7030100740 2023-07-14T18:11:35.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703019d970 con 0x7f7030100740 2023-07-14T18:11:35.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f7026ffd700 1 -- 172.21.15.115:0/2454787991 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280115e0 con 0x7f7030100740 2023-07-14T18:11:35.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.799+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703019de20 con 0x7f7030100740 2023-07-14T18:11:35.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.800+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703010a220 con 0x7f7030100740 2023-07-14T18:11:35.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.801+0000 7f7026ffd700 1 -- 172.21.15.115:0/2454787991 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f7028012420 con 0x7f7030100740 2023-07-14T18:11:35.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.802+0000 7f7026ffd700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f701c0a11e0 0x7f701c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:35.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.802+0000 7f7035ccd700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f701c0a11e0 0x7f701c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:35.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.802+0000 7f7026ffd700 1 -- 172.21.15.115:0/2454787991 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f70280ccf80 con 0x7f7030100740 2023-07-14T18:11:35.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.802+0000 7f7035ccd700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f701c0a11e0 0x7f701c0a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7020005fd0 tx=0x7f7020005dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:35.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:35.806+0000 7f7026ffd700 1 -- 172.21.15.115:0/2454787991 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7028096310 con 0x7f7030100740 2023-07-14T18:11:36.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.007+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7030068890 con 0x7f701c0a11e0 2023-07-14T18:11:36.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.008+0000 7f7026ffd700 1 -- 172.21.15.115:0/2454787991 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7030068890 con 0x7f701c0a11e0 2023-07-14T18:11:36.048 INFO:teuthology.orchestra.run.smithi115.stdout:Backend: cephadm 2023-07-14T18:11:36.048 INFO:teuthology.orchestra.run.smithi115.stdout:Available: Yes 2023-07-14T18:11:36.049 INFO:teuthology.orchestra.run.smithi115.stdout:Paused: No 2023-07-14T18:11:36.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f701c0a11e0 msgr2=0x7f701c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:36.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f701c0a11e0 0x7f701c0a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7020005fd0 tx=0x7f7020005dd0 comp rx=0 tx=0).stop 2023-07-14T18:11:36.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7030100740 msgr2=0x7f703019cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:36.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7030100740 0x7f703019cdf0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7028007da0 tx=0x7f702800a340 comp rx=0 tx=0).stop 2023-07-14T18:11:36.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 shutdown_connections 2023-07-14T18:11:36.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f701c0a11e0 0x7f701c0a3690 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:36.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7030100740 0x7f703019cdf0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:36.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70300febb0 0x7f7030195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:36.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 --2- 172.21.15.115:0/2454787991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70300ffdb0 0x7f7030195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:36.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 >> 172.21.15.115:0/2454787991 conn(0x7f70300fa1e0 msgr2=0x7f7030068180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:36.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 shutdown_connections 2023-07-14T18:11:36.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:36.011+0000 7f7037f31700 1 -- 172.21.15.115:0/2454787991 wait complete. 2023-07-14T18:11:36.356 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:36 smithi115 ceph-mon[127451]: pgmap v292: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2023-07-14T18:11:36.477 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ps' 2023-07-14T18:11:36.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:36 smithi150 ceph-mon[128853]: pgmap v292: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2023-07-14T18:11:36.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:36 smithi187 ceph-mon[129317]: pgmap v292: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2023-07-14T18:11:37.028 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:37.504 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:37 smithi115 ceph-mon[127451]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:37.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:37 smithi150 ceph-mon[128853]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:37.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:37 smithi187 ceph-mon[129317]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 -- 172.21.15.115:0/161630226 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 msgr2=0x7f15b0100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 --2- 172.21.15.115:0/161630226 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 0x7f15b0100200 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009990 tx=0x7f15a0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 -- 172.21.15.115:0/161630226 shutdown_connections 2023-07-14T18:11:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 --2- 172.21.15.115:0/161630226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b0100740 0x7f15b0107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 --2- 172.21.15.115:0/161630226 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15b00febb0 0x7f15b00fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 --2- 172.21.15.115:0/161630226 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 0x7f15b0100200 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 -- 172.21.15.115:0/161630226 >> 172.21.15.115:0/161630226 conn(0x7f15b00fa1e0 msgr2=0x7f15b00fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 -- 172.21.15.115:0/161630226 shutdown_connections 2023-07-14T18:11:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 -- 172.21.15.115:0/161630226 wait complete. 2023-07-14T18:11:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 Processor -- start 2023-07-14T18:11:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.434+0000 7f15b7758700 1 -- start start 2023-07-14T18:11:38.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b7758700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15b00febb0 0x7f15b00694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b7758700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 0x7f15b019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b7758700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b0100740 0x7f15b019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b7758700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b019d420 con 0x7f15b00ffdb0 2023-07-14T18:11:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b7758700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b019d590 con 0x7f15b0100740 2023-07-14T18:11:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b7758700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b019d750 con 0x7f15b00febb0 2023-07-14T18:11:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b4cf3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 0x7f15b019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b4cf3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 0x7f15b019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53902/0 (socket says 172.21.15.115:53902) 2023-07-14T18:11:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b4cf3700 1 -- 172.21.15.115:0/3134368575 learned_addr learned my addr 172.21.15.115:0/3134368575 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b54f4700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15b00febb0 0x7f15b00694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b5cf5700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b0100740 0x7f15b019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b4cf3700 1 -- 172.21.15.115:0/3134368575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15b00febb0 msgr2=0x7f15b00694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b4cf3700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15b00febb0 0x7f15b00694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b4cf3700 1 -- 172.21.15.115:0/3134368575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b0100740 msgr2=0x7f15b019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b4cf3700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b0100740 0x7f15b019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b4cf3700 1 -- 172.21.15.115:0/3134368575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a00095f0 con 0x7f15b00ffdb0 2023-07-14T18:11:38.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b5cf5700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b0100740 0x7f15b019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b4cf3700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 0x7f15b019a640 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f15a00059d0 tx=0x7f15a0004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15a67fc700 1 -- 172.21.15.115:0/3134368575 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a001d070 con 0x7f15b00ffdb0 2023-07-14T18:11:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b019d9d0 con 0x7f15b00ffdb0 2023-07-14T18:11:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15a67fc700 1 -- 172.21.15.115:0/3134368575 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f15a000f460 con 0x7f15b00ffdb0 2023-07-14T18:11:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.435+0000 7f15a67fc700 1 -- 172.21.15.115:0/3134368575 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a0021670 con 0x7f15b00ffdb0 2023-07-14T18:11:38.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.436+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b019de70 con 0x7f15b00ffdb0 2023-07-14T18:11:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.436+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b010a220 con 0x7f15b00ffdb0 2023-07-14T18:11:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.439+0000 7f15a67fc700 1 -- 172.21.15.115:0/3134368575 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f15a000f5d0 con 0x7f15b00ffdb0 2023-07-14T18:11:38.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.439+0000 7f15a67fc700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f159c0a11e0 0x7f159c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.439+0000 7f15b54f4700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f159c0a11e0 0x7f159c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.439+0000 7f15a67fc700 1 -- 172.21.15.115:0/3134368575 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f15a00ce470 con 0x7f15b00ffdb0 2023-07-14T18:11:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.439+0000 7f15b54f4700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f159c0a11e0 0x7f159c0a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f15b00ffc10 tx=0x7f15ac00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:38.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.443+0000 7f15a67fc700 1 -- 172.21.15.115:0/3134368575 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f15a0097780 con 0x7f15b00ffdb0 2023-07-14T18:11:38.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:38 smithi115 ceph-mon[127451]: pgmap v293: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2023-07-14T18:11:38.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:38 smithi150 ceph-mon[128853]: pgmap v293: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2023-07-14T18:11:38.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.640+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f15b0060a80 con 0x7f159c0a11e0 2023-07-14T18:11:38.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:38 smithi187 ceph-mon[129317]: pgmap v293: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2023-07-14T18:11:38.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.652+0000 7f15a67fc700 1 -- 172.21.15.115:0/3134368575 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f15b0060a80 con 0x7f159c0a11e0 2023-07-14T18:11:38.692 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-14T18:11:38.692 INFO:teuthology.orchestra.run.smithi115.stdout:mds.a smithi115 running (89s) - 89s 15.0M - 18.0.0-4940-g81c9387d 219b180907a8 85d35ede4f28 2023-07-14T18:11:38.692 INFO:teuthology.orchestra.run.smithi115.stdout:mds.b smithi150 running (87s) - 87s 30.2M - 18.0.0-4940-g81c9387d 219b180907a8 e0dc4bcaca79 2023-07-14T18:11:38.692 INFO:teuthology.orchestra.run.smithi115.stdout:mds.c smithi187 running (98s) - 98s 20.1M - 18.0.0-4940-g81c9387d 219b180907a8 c46fafde96e7 2023-07-14T18:11:38.693 INFO:teuthology.orchestra.run.smithi115.stdout:mds.d smithi115 running (104s) - 104s 16.5M - 18.0.0-4940-g81c9387d 219b180907a8 093c77f671f5 2023-07-14T18:11:38.693 INFO:teuthology.orchestra.run.smithi115.stdout:mds.e smithi150 running (102s) - 102s 16.6M - 18.0.0-4940-g81c9387d 219b180907a8 65c51609c3d9 2023-07-14T18:11:38.693 INFO:teuthology.orchestra.run.smithi115.stdout:mds.f smithi187 running (100s) - 100s 18.5M - 18.0.0-4940-g81c9387d 219b180907a8 8fe40feddcf9 2023-07-14T18:11:38.693 INFO:teuthology.orchestra.run.smithi115.stdout:mds.g smithi115 running (91s) - 91s 15.1M - 18.0.0-4940-g81c9387d 219b180907a8 a7f80b32166e 2023-07-14T18:11:38.693 INFO:teuthology.orchestra.run.smithi115.stdout:mds.h smithi150 running (96s) - 96s 20.2M - 18.0.0-4940-g81c9387d 219b180907a8 8f54d8cfb00c 2023-07-14T18:11:38.693 INFO:teuthology.orchestra.run.smithi115.stdout:mds.i smithi187 running (94s) - 93s 18.8M - 18.0.0-4940-g81c9387d 219b180907a8 f2068be83577 2023-07-14T18:11:38.693 INFO:teuthology.orchestra.run.smithi115.stdout:mds.j smithi115 running (107s) - 106s 16.3M - 18.0.0-4940-g81c9387d 219b180907a8 ab53730f3665 2023-07-14T18:11:38.694 INFO:teuthology.orchestra.run.smithi115.stdout:mds.k smithi150 running (109s) - 108s 16.5M - 18.0.0-4940-g81c9387d 219b180907a8 be054a948245 2023-07-14T18:11:38.694 INFO:teuthology.orchestra.run.smithi115.stdout:mds.l smithi187 running (111s) - 110s 17.5M - 18.0.0-4940-g81c9387d 219b180907a8 145fe675c3d5 2023-07-14T18:11:38.694 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi115 *:9283,8765 running (7m) - 7m 518M - 18.0.0-4940-g81c9387d 219b180907a8 8b6d3c80ebc2 2023-07-14T18:11:38.694 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi150 *:8443,8765 running (6m) - 6m 448M - 18.0.0-4940-g81c9387d 219b180907a8 28b0dc3744a8 2023-07-14T18:11:38.694 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.z smithi187 *:8443,8765 running (6m) - 6m 447M - 18.0.0-4940-g81c9387d 219b180907a8 268066ecc647 2023-07-14T18:11:38.694 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (7m) - 8m 95.7M 2048M 18.0.0-4940-g81c9387d 219b180907a8 89625cb3df65 2023-07-14T18:11:38.694 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi150 running (6m) - 6m 58.8M 2048M 18.0.0-4940-g81c9387d 219b180907a8 8bad80ddd876 2023-07-14T18:11:38.695 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi187 running (6m) - 6m 61.2M 2048M 18.0.0-4940-g81c9387d 219b180907a8 8bb9a2af086a 2023-07-14T18:11:38.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5m) - 5m 69.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 8c5241101b1c 2023-07-14T18:11:38.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi150 running (5m) - 5m 77.3M 4292M 18.0.0-4940-g81c9387d 219b180907a8 981df25731a0 2023-07-14T18:11:38.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi187 running (5m) - 5m 76.5M 4292M 18.0.0-4940-g81c9387d 219b180907a8 027daf3fa96c 2023-07-14T18:11:38.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4m) - 4m 65.0M 4292M 18.0.0-4940-g81c9387d 219b180907a8 6cebf24344c1 2023-07-14T18:11:38.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi150 running (4m) - 4m 63.0M 4292M 18.0.0-4940-g81c9387d 219b180907a8 74a2a904f0ee 2023-07-14T18:11:38.695 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi187 running (4m) - 4m 70.2M 4292M 18.0.0-4940-g81c9387d 219b180907a8 f25f5b7a2be6 2023-07-14T18:11:38.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi115 running (3m) - 3m 61.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 2ec83a6e9f14 2023-07-14T18:11:38.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi150 running (3m) - 3m 67.7M 4292M 18.0.0-4940-g81c9387d 219b180907a8 9102c86e353b 2023-07-14T18:11:38.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.8 smithi187 running (3m) - 3m 62.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 4035143702c7 2023-07-14T18:11:38.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.9 smithi115 running (2m) - 2m 59.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 1f82b50d7fb5 2023-07-14T18:11:38.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.10 smithi150 running (2m) - 2m 62.6M 4292M 18.0.0-4940-g81c9387d 219b180907a8 b1a734304cb3 2023-07-14T18:11:38.696 INFO:teuthology.orchestra.run.smithi115.stdout:osd.11 smithi187 running (2m) - 2m 57.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 30f579c87b8f 2023-07-14T18:11:38.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f159c0a11e0 msgr2=0x7f159c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:38.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f159c0a11e0 0x7f159c0a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f15b00ffc10 tx=0x7f15ac00b410 comp rx=0 tx=0).stop 2023-07-14T18:11:38.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 msgr2=0x7f15b019a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:38.697 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 0x7f15b019a640 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f15a00059d0 tx=0x7f15a0004790 comp rx=0 tx=0).stop 2023-07-14T18:11:38.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 shutdown_connections 2023-07-14T18:11:38.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f159c0a11e0 0x7f159c0a3690 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b0100740 0x7f15b019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15b00febb0 0x7f15b00694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 --2- 172.21.15.115:0/3134368575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15b00ffdb0 0x7f15b019a640 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:38.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 >> 172.21.15.115:0/3134368575 conn(0x7f15b00fa1e0 msgr2=0x7f15b00fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:38.698 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 shutdown_connections 2023-07-14T18:11:38.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:38.654+0000 7f15b7758700 1 -- 172.21.15.115:0/3134368575 wait complete. 2023-07-14T18:11:39.190 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ls' 2023-07-14T18:11:39.745 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:40.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:40 smithi115 ceph-mon[127451]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:40.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:40 smithi115 ceph-mon[127451]: pgmap v294: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 14 B/s, 1 keys/s, 0 objects/s recovering 2023-07-14T18:11:40.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:40 smithi150 ceph-mon[128853]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:40.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:40 smithi150 ceph-mon[128853]: pgmap v294: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 14 B/s, 1 keys/s, 0 objects/s recovering 2023-07-14T18:11:40.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:40 smithi187 ceph-mon[129317]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:40.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:40 smithi187 ceph-mon[129317]: pgmap v294: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 14 B/s, 1 keys/s, 0 objects/s recovering 2023-07-14T18:11:41.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 -- 172.21.15.115:0/2492537041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 msgr2=0x7fe77c0fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:41.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 --2- 172.21.15.115:0/2492537041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 0x7fe77c0fefc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009990 tx=0x7fe76c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 -- 172.21.15.115:0/2492537041 shutdown_connections 2023-07-14T18:11:41.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 --2- 172.21.15.115:0/2492537041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77c100740 0x7fe77c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 --2- 172.21.15.115:0/2492537041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 0x7fe77c0fefc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 --2- 172.21.15.115:0/2492537041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe77c0ffdb0 0x7fe77c100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 -- 172.21.15.115:0/2492537041 >> 172.21.15.115:0/2492537041 conn(0x7fe77c0fa1e0 msgr2=0x7fe77c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:41.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 -- 172.21.15.115:0/2492537041 shutdown_connections 2023-07-14T18:11:41.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 -- 172.21.15.115:0/2492537041 wait complete. 2023-07-14T18:11:41.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 Processor -- start 2023-07-14T18:11:41.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.265+0000 7fe783354700 1 -- start start 2023-07-14T18:11:41.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe783354700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 0x7fe77c068c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:41.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe783354700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77c0ffdb0 0x7fe77c19a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:41.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe783354700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe77c100740 0x7fe77c19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:41.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe783354700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c19d370 con 0x7fe77c0febb0 2023-07-14T18:11:41.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe783354700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c19d4e0 con 0x7fe77c0ffdb0 2023-07-14T18:11:41.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe783354700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77c19d6a0 con 0x7fe77c100740 2023-07-14T18:11:41.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7810f0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 0x7fe77c068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:41.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7810f0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 0x7fe77c068c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53914/0 (socket says 172.21.15.115:53914) 2023-07-14T18:11:41.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7810f0700 1 -- 172.21.15.115:0/1891065263 learned_addr learned my addr 172.21.15.115:0/1891065263 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:41.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7808ef700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77c0ffdb0 0x7fe77c19a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:41.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7810f0700 1 -- 172.21.15.115:0/1891065263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe77c100740 msgr2=0x7fe77c19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:41.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7810f0700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe77c100740 0x7fe77c19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7810f0700 1 -- 172.21.15.115:0/1891065263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77c0ffdb0 msgr2=0x7fe77c19a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:41.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7810f0700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77c0ffdb0 0x7fe77c19a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7810f0700 1 -- 172.21.15.115:0/1891065263 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c0095f0 con 0x7fe77c0febb0 2023-07-14T18:11:41.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7808ef700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77c0ffdb0 0x7fe77c19a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:41.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7810f0700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 0x7fe77c068c80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe76c004730 tx=0x7fe76c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:41.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7727fc700 1 -- 172.21.15.115:0/1891065263 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c01d070 con 0x7fe77c0febb0 2023-07-14T18:11:41.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7727fc700 1 -- 172.21.15.115:0/1891065263 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe76c004810 con 0x7fe77c0febb0 2023-07-14T18:11:41.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77c19d920 con 0x7fe77c0febb0 2023-07-14T18:11:41.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe7727fc700 1 -- 172.21.15.115:0/1891065263 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c0052d0 con 0x7fe77c0febb0 2023-07-14T18:11:41.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.266+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77c19dd30 con 0x7fe77c0febb0 2023-07-14T18:11:41.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.267+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77c1061b0 con 0x7fe77c0febb0 2023-07-14T18:11:41.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.268+0000 7fe7727fc700 1 -- 172.21.15.115:0/1891065263 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fe76c005490 con 0x7fe77c0febb0 2023-07-14T18:11:41.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.269+0000 7fe7727fc700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7680a11e0 0x7fe7680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:41.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.269+0000 7fe7808ef700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7680a11e0 0x7fe7680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:41.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.269+0000 7fe7727fc700 1 -- 172.21.15.115:0/1891065263 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe76c0ce040 con 0x7fe77c0febb0 2023-07-14T18:11:41.311 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.269+0000 7fe7808ef700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7680a11e0 0x7fe7680a3690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe778005fd0 tx=0x7fe778005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:41.312 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.273+0000 7fe7727fc700 1 -- 172.21.15.115:0/1891065263 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe76c0973b0 con 0x7fe77c0febb0 2023-07-14T18:11:41.508 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.469+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe77c060a80 con 0x7fe7680a11e0 2023-07-14T18:11:41.513 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.474+0000 7fe7727fc700 1 -- 172.21.15.115:0/1891065263 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+985 (secure 0 0 0) 0x7fe77c060a80 con 0x7fe7680a11e0 2023-07-14T18:11:41.513 INFO:teuthology.orchestra.run.smithi115.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-07-14T18:11:41.513 INFO:teuthology.orchestra.run.smithi115.stdout:mds.all 12/12 - 115s smithi115=a;smithi115=d;smithi115=g;smithi115=j;smithi150=b;smithi150=e;smithi150=h;smithi150=k;smithi187=c;smithi187=f;smithi187=i;smithi187=l;count:12 2023-07-14T18:11:41.514 INFO:teuthology.orchestra.run.smithi115.stdout:mgr 3/3 - 6m smithi115=x;smithi150=y;smithi187=z;count:3 2023-07-14T18:11:41.514 INFO:teuthology.orchestra.run.smithi115.stdout:mon 3/3 - 6m smithi115:172.21.15.115=a;smithi150:172.21.15.150=b;smithi187:172.21.15.187=c;count:3 2023-07-14T18:11:41.514 INFO:teuthology.orchestra.run.smithi115.stdout:osd 12 - - 2023-07-14T18:11:41.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7680a11e0 msgr2=0x7fe7680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:41.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7680a11e0 0x7fe7680a3690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe778005fd0 tx=0x7fe778005e30 comp rx=0 tx=0).stop 2023-07-14T18:11:41.515 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 msgr2=0x7fe77c068c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:41.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 0x7fe77c068c80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe76c004730 tx=0x7fe76c003820 comp rx=0 tx=0).stop 2023-07-14T18:11:41.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 shutdown_connections 2023-07-14T18:11:41.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7680a11e0 0x7fe7680a3690 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.516 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe77c100740 0x7fe77c19cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe77c0febb0 0x7fe77c068c80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 --2- 172.21.15.115:0/1891065263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe77c0ffdb0 0x7fe77c19a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:41.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 >> 172.21.15.115:0/1891065263 conn(0x7fe77c0fa1e0 msgr2=0x7fe77c1032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:41.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 shutdown_connections 2023-07-14T18:11:41.517 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:41.476+0000 7fe783354700 1 -- 172.21.15.115:0/1891065263 wait complete. 2023-07-14T18:11:42.008 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- bash -c 'ceph orch host ls' 2023-07-14T18:11:42.559 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:42.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:42 smithi115 ceph-mon[127451]: pgmap v295: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 13 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T18:11:42.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:42 smithi150 ceph-mon[128853]: pgmap v295: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 13 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T18:11:42.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:42 smithi187 ceph-mon[129317]: pgmap v295: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 13 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T18:11:43.454 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:43 smithi115 ceph-mon[127451]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:43.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:43 smithi150 ceph-mon[128853]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:43.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:43 smithi187 ceph-mon[129317]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:43.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.914+0000 7fdd95487700 1 -- 172.21.15.115:0/1121828034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd900fee10 msgr2=0x7fdd900ff280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.914+0000 7fdd95487700 1 --2- 172.21.15.115:0/1121828034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd900fee10 0x7fdd900ff280 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009990 tx=0x7fdd80009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.914+0000 7fdd95487700 1 -- 172.21.15.115:0/1121828034 shutdown_connections 2023-07-14T18:11:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.914+0000 7fdd95487700 1 --2- 172.21.15.115:0/1121828034 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd900ff7c0 0x7fdd90107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.914+0000 7fdd95487700 1 --2- 172.21.15.115:0/1121828034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd90100240 0x7fdd90100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:43.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.914+0000 7fdd95487700 1 --2- 172.21.15.115:0/1121828034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd900fee10 0x7fdd900ff280 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.914+0000 7fdd95487700 1 -- 172.21.15.115:0/1121828034 >> 172.21.15.115:0/1121828034 conn(0x7fdd900f9990 msgr2=0x7fdd900fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.914+0000 7fdd95487700 1 -- 172.21.15.115:0/1121828034 shutdown_connections 2023-07-14T18:11:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.914+0000 7fdd95487700 1 -- 172.21.15.115:0/1121828034 wait complete. 2023-07-14T18:11:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.915+0000 7fdd95487700 1 Processor -- start 2023-07-14T18:11:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.915+0000 7fdd95487700 1 -- start start 2023-07-14T18:11:43.956 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.915+0000 7fdd95487700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd900fee10 0x7fdd90197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.915+0000 7fdd95487700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd900ff7c0 0x7fdd90197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.915+0000 7fdd95487700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90100240 0x7fdd9019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.915+0000 7fdd95487700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9019f5f0 con 0x7fdd90100240 2023-07-14T18:11:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.915+0000 7fdd95487700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9019f730 con 0x7fdd900fee10 2023-07-14T18:11:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.915+0000 7fdd95487700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9019f8c0 con 0x7fdd900ff7c0 2023-07-14T18:11:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90100240 0x7fdd9019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:43.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90100240 0x7fdd9019f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53930/0 (socket says 172.21.15.115:53930) 2023-07-14T18:11:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8f7fe700 1 -- 172.21.15.115:0/1169001531 learned_addr learned my addr 172.21.15.115:0/1169001531 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8e7fc700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd900ff7c0 0x7fdd90197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8effd700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd900fee10 0x7fdd90197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8f7fe700 1 -- 172.21.15.115:0/1169001531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd900ff7c0 msgr2=0x7fdd90197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8f7fe700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd900ff7c0 0x7fdd90197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8f7fe700 1 -- 172.21.15.115:0/1169001531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd900fee10 msgr2=0x7fdd90197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8f7fe700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd900fee10 0x7fdd90197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:43.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8f7fe700 1 -- 172.21.15.115:0/1169001531 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd800095f0 con 0x7fdd90100240 2023-07-14T18:11:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd8f7fe700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90100240 0x7fdd9019f020 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdd88007f10 tx=0x7fdd8800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd77fff700 1 -- 172.21.15.115:0/1169001531 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88017070 con 0x7fdd90100240 2023-07-14T18:11:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9019fba0 con 0x7fdd90100240 2023-07-14T18:11:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd77fff700 1 -- 172.21.15.115:0/1169001531 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdd8800f040 con 0x7fdd90100240 2023-07-14T18:11:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd77fff700 1 -- 172.21.15.115:0/1169001531 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88005710 con 0x7fdd90100240 2023-07-14T18:11:43.959 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.916+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd901a00a0 con 0x7fdd90100240 2023-07-14T18:11:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.918+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9010a280 con 0x7fdd90100240 2023-07-14T18:11:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.918+0000 7fdd77fff700 1 -- 172.21.15.115:0/1169001531 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fdd88004ad0 con 0x7fdd90100240 2023-07-14T18:11:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.919+0000 7fdd77fff700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd780a1260 0x7fdd780a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:43.960 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.919+0000 7fdd8effd700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd780a1260 0x7fdd780a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.919+0000 7fdd77fff700 1 -- 172.21.15.115:0/1169001531 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdd880cc550 con 0x7fdd90100240 2023-07-14T18:11:43.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.919+0000 7fdd8effd700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd780a1260 0x7fdd780a3710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdd84005ea0 tx=0x7fdd84005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:43.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:43.923+0000 7fdd77fff700 1 -- 172.21.15.115:0/1169001531 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdd88095830 con 0x7fdd90100240 2023-07-14T18:11:44.155 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.116+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd900fe510 con 0x7fdd780a1260 2023-07-14T18:11:44.156 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.117+0000 7fdd77fff700 1 -- 172.21.15.115:0/1169001531 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fdd900fe510 con 0x7fdd780a1260 2023-07-14T18:11:44.156 INFO:teuthology.orchestra.run.smithi115.stdout:HOST ADDR LABELS STATUS 2023-07-14T18:11:44.156 INFO:teuthology.orchestra.run.smithi115.stdout:smithi115 172.21.15.115 2023-07-14T18:11:44.157 INFO:teuthology.orchestra.run.smithi115.stdout:smithi150 172.21.15.150 2023-07-14T18:11:44.157 INFO:teuthology.orchestra.run.smithi115.stdout:smithi187 172.21.15.187 2023-07-14T18:11:44.157 INFO:teuthology.orchestra.run.smithi115.stdout:3 hosts in cluster 2023-07-14T18:11:44.158 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd780a1260 msgr2=0x7fdd780a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:44.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd780a1260 0x7fdd780a3710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdd84005ea0 tx=0x7fdd84005e30 comp rx=0 tx=0).stop 2023-07-14T18:11:44.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90100240 msgr2=0x7fdd9019f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:44.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90100240 0x7fdd9019f020 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdd88007f10 tx=0x7fdd8800d3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:44.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 shutdown_connections 2023-07-14T18:11:44.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd780a1260 0x7fdd780a3710 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:44.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd90100240 0x7fdd9019f020 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:44.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd900fee10 0x7fdd90197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:44.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 --2- 172.21.15.115:0/1169001531 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd900ff7c0 0x7fdd90197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:44.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.119+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 >> 172.21.15.115:0/1169001531 conn(0x7fdd900f9990 msgr2=0x7fdd900fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:44.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.120+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 shutdown_connections 2023-07-14T18:11:44.160 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:44.120+0000 7fdd95487700 1 -- 172.21.15.115:0/1169001531 wait complete. 2023-07-14T18:11:44.606 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:44 smithi115 ceph-mon[127451]: pgmap v296: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 12 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T18:11:44.607 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- bash -c 'ceph orch device ls' 2023-07-14T18:11:44.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:44 smithi150 ceph-mon[128853]: pgmap v296: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 12 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T18:11:44.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:44 smithi187 ceph-mon[129317]: pgmap v296: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 12 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T18:11:45.158 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:45.576 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:45 smithi115 ceph-mon[127451]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:45.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:45 smithi150 ceph-mon[128853]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:45.684 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:45 smithi187 ceph-mon[129317]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:46.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:46 smithi115 ceph-mon[127451]: pgmap v297: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T18:11:46.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:46 smithi150 ceph-mon[128853]: pgmap v297: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T18:11:46.631 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.592+0000 7f300da5f700 1 -- 172.21.15.115:0/2534986589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3008100f50 msgr2=0x7f30081013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:46.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.592+0000 7f300da5f700 1 --2- 172.21.15.115:0/2534986589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3008100f50 0x7f30081013a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009990 tx=0x7f2ff0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.592+0000 7f300da5f700 1 -- 172.21.15.115:0/2534986589 shutdown_connections 2023-07-14T18:11:46.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.592+0000 7f300da5f700 1 --2- 172.21.15.115:0/2534986589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30081018e0 0x7f30081081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.592+0000 7f300da5f700 1 --2- 172.21.15.115:0/2534986589 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30080ffd50 0x7f3008100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.592+0000 7f300da5f700 1 --2- 172.21.15.115:0/2534986589 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3008100f50 0x7f30081013a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.632 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.592+0000 7f300da5f700 1 -- 172.21.15.115:0/2534986589 >> 172.21.15.115:0/2534986589 conn(0x7f30080fb380 msgr2=0x7f30080fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:46.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.592+0000 7f300da5f700 1 -- 172.21.15.115:0/2534986589 shutdown_connections 2023-07-14T18:11:46.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.592+0000 7f300da5f700 1 -- 172.21.15.115:0/2534986589 wait complete. 2023-07-14T18:11:46.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f300da5f700 1 Processor -- start 2023-07-14T18:11:46.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f300da5f700 1 -- start start 2023-07-14T18:11:46.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f300da5f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f30080ffd50 0x7f3008195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:46.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f300da5f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3008100f50 0x7f3008195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:46.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f300da5f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30081018e0 0x7f300819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:46.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f300da5f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300819d3e0 con 0x7f30080ffd50 2023-07-14T18:11:46.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f300da5f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300819d520 con 0x7f3008100f50 2023-07-14T18:11:46.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f300da5f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300819d6b0 con 0x7f30081018e0 2023-07-14T18:11:46.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f3006ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f30080ffd50 0x7f3008195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:46.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f3006ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f30080ffd50 0x7f3008195850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53938/0 (socket says 172.21.15.115:53938) 2023-07-14T18:11:46.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f3006ffd700 1 -- 172.21.15.115:0/1638441472 learned_addr learned my addr 172.21.15.115:0/1638441472 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:46.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.593+0000 7f30067fc700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3008100f50 0x7f3008195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:46.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f30077fe700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30081018e0 0x7f300819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:46.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f3006ffd700 1 -- 172.21.15.115:0/1638441472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30081018e0 msgr2=0x7f300819ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:46.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f3006ffd700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30081018e0 0x7f300819ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f3006ffd700 1 -- 172.21.15.115:0/1638441472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3008100f50 msgr2=0x7f3008195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:46.636 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f3006ffd700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3008100f50 0x7f3008195d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f3006ffd700 1 -- 172.21.15.115:0/1638441472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff00095f0 con 0x7f30080ffd50 2023-07-14T18:11:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f3006ffd700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f30080ffd50 0x7f3008195850 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c9c0 tx=0x7f2ff800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f300ca5d700 1 -- 172.21.15.115:0/1638441472 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8007ac0 con 0x7f30080ffd50 2023-07-14T18:11:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f300ca5d700 1 -- 172.21.15.115:0/1638441472 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2ff8007c20 con 0x7f30080ffd50 2023-07-14T18:11:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300819d990 con 0x7f30080ffd50 2023-07-14T18:11:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f300ca5d700 1 -- 172.21.15.115:0/1638441472 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8018660 con 0x7f30080ffd50 2023-07-14T18:11:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.594+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300819de90 con 0x7f30080ffd50 2023-07-14T18:11:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.595+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3008107340 con 0x7f30080ffd50 2023-07-14T18:11:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.597+0000 7f300ca5d700 1 -- 172.21.15.115:0/1638441472 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f2ff800f450 con 0x7f30080ffd50 2023-07-14T18:11:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.598+0000 7f300ca5d700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ff40a11e0 0x7f2ff40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.598+0000 7f30067fc700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ff40a11e0 0x7f2ff40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:46.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.598+0000 7f300ca5d700 1 -- 172.21.15.115:0/1638441472 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2ff80cce20 con 0x7f30080ffd50 2023-07-14T18:11:46.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.598+0000 7f30067fc700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ff40a11e0 0x7f2ff40a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000b5c0 tx=0x7f2ff0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:46.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.601+0000 7f300ca5d700 1 -- 172.21.15.115:0/1638441472 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2ff8096190 con 0x7f30080ffd50 2023-07-14T18:11:46.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:46 smithi187 ceph-mon[129317]: pgmap v297: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T18:11:46.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.794+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3008060a80 con 0x7f2ff40a11e0 2023-07-14T18:11:46.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.795+0000 7f300ca5d700 1 -- 172.21.15.115:0/1638441472 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f3008060a80 con 0x7f2ff40a11e0 2023-07-14T18:11:46.835 INFO:teuthology.orchestra.run.smithi115.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-07-14T18:11:46.835 INFO:teuthology.orchestra.run.smithi115.stdout:smithi115 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UU400BGN 372G No 2m ago LVM detected, locked 2023-07-14T18:11:46.835 INFO:teuthology.orchestra.run.smithi115.stdout:smithi150 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GX400BGN 372G No 2m ago LVM detected, locked 2023-07-14T18:11:46.835 INFO:teuthology.orchestra.run.smithi115.stdout:smithi187 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500MW400BGN 372G No 2m ago LVM detected, locked 2023-07-14T18:11:46.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ff40a11e0 msgr2=0x7f2ff40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:46.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ff40a11e0 0x7f2ff40a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000b5c0 tx=0x7f2ff0005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f30080ffd50 msgr2=0x7f3008195850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:46.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f30080ffd50 0x7f3008195850 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c9c0 tx=0x7f2ff800ccd0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 shutdown_connections 2023-07-14T18:11:46.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2ff40a11e0 0x7f2ff40a3690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30081018e0 0x7f300819ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f30080ffd50 0x7f3008195850 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 --2- 172.21.15.115:0/1638441472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3008100f50 0x7f3008195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:46.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 >> 172.21.15.115:0/1638441472 conn(0x7f30080fb380 msgr2=0x7f3008104480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:46.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 shutdown_connections 2023-07-14T18:11:46.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:46.798+0000 7f300da5f700 1 -- 172.21.15.115:0/1638441472 wait complete. 2023-07-14T18:11:47.311 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T18:11:47.321 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi115.front.sepia.ceph.com 2023-07-14T18:11:47.321 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- bash -c 'ceph fs dump' 2023-07-14T18:11:47.873 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:48.432 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:48 smithi115 ceph-mon[127451]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:48.432 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:48 smithi115 ceph-mon[127451]: pgmap v298: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T18:11:48.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:48 smithi150 ceph-mon[128853]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:48.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:48 smithi150 ceph-mon[128853]: pgmap v298: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T18:11:48.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:48 smithi187 ceph-mon[129317]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:11:48.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:48 smithi187 ceph-mon[129317]: pgmap v298: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T18:11:49.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.202+0000 7fa1d19c6700 1 -- 172.21.15.115:0/51143468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc0ff810 msgr2=0x7fa1cc1070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:49.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.202+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/51143468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc0ff810 0x7fa1cc1070c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400b0a0 tx=0x7fa1c400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.203+0000 7fa1d19c6700 1 -- 172.21.15.115:0/51143468 shutdown_connections 2023-07-14T18:11:49.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.203+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/51143468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc0ff810 0x7fa1cc1070c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.203+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/51143468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1cc100290 0x7fa1cc1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.203+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/51143468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1cc0fee60 0x7fa1cc0ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.203+0000 7fa1d19c6700 1 -- 172.21.15.115:0/51143468 >> 172.21.15.115:0/51143468 conn(0x7fa1cc0f99c0 msgr2=0x7fa1cc0fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:49.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.203+0000 7fa1d19c6700 1 -- 172.21.15.115:0/51143468 shutdown_connections 2023-07-14T18:11:49.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.203+0000 7fa1d19c6700 1 -- 172.21.15.115:0/51143468 wait complete. 2023-07-14T18:11:49.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.203+0000 7fa1d19c6700 1 Processor -- start 2023-07-14T18:11:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1d19c6700 1 -- start start 2023-07-14T18:11:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1d19c6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1cc0fee60 0x7fa1cc1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1d19c6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1cc0ff810 0x7fa1cc195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1d19c6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc100290 0x7fa1cc19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1d19c6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc19d530 con 0x7fa1cc100290 2023-07-14T18:11:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1d19c6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc19d670 con 0x7fa1cc0ff810 2023-07-14T18:11:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1d19c6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc19d800 con 0x7fa1cc0fee60 2023-07-14T18:11:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1cb7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc100290 0x7fa1cc19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1cb7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc100290 0x7fa1cc19cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53954/0 (socket says 172.21.15.115:53954) 2023-07-14T18:11:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1cb7fe700 1 -- 172.21.15.115:0/2276887041 learned_addr learned my addr 172.21.15.115:0/2276887041 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1caffd700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1cc0fee60 0x7fa1cc1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1ca7fc700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1cc0ff810 0x7fa1cc195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1cb7fe700 1 -- 172.21.15.115:0/2276887041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1cc0fee60 msgr2=0x7fa1cc1959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1cb7fe700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1cc0fee60 0x7fa1cc1959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1cb7fe700 1 -- 172.21.15.115:0/2276887041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1cc0ff810 msgr2=0x7fa1cc195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1cb7fe700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1cc0ff810 0x7fa1cc195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1cb7fe700 1 -- 172.21.15.115:0/2276887041 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c4009d00 con 0x7fa1cc100290 2023-07-14T18:11:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1cb7fe700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc100290 0x7fa1cc19cf60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4006070 tx=0x7fa1c4004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.204+0000 7fa1d09c4700 1 -- 172.21.15.115:0/2276887041 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c400e040 con 0x7fa1cc100290 2023-07-14T18:11:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.205+0000 7fa1d09c4700 1 -- 172.21.15.115:0/2276887041 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa1c4003e90 con 0x7fa1cc100290 2023-07-14T18:11:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.205+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1cc107a70 con 0x7fa1cc100290 2023-07-14T18:11:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.205+0000 7fa1d09c4700 1 -- 172.21.15.115:0/2276887041 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4004020 con 0x7fa1cc100290 2023-07-14T18:11:49.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.205+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1cc107f70 con 0x7fa1cc100290 2023-07-14T18:11:49.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.205+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1cc10b4b0 con 0x7fa1cc100290 2023-07-14T18:11:49.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.207+0000 7fa1d09c4700 1 -- 172.21.15.115:0/2276887041 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fa1c401b660 con 0x7fa1cc100290 2023-07-14T18:11:49.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.207+0000 7fa1d09c4700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa1b40a11e0 0x7fa1b40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:49.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.207+0000 7fa1caffd700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa1b40a11e0 0x7fa1b40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:49.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.207+0000 7fa1d09c4700 1 -- 172.21.15.115:0/2276887041 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa1c40ceb60 con 0x7fa1cc100290 2023-07-14T18:11:49.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.208+0000 7fa1caffd700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa1b40a11e0 0x7fa1b40a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc005950 tx=0x7fa1bc0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:49.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.211+0000 7fa1d09c4700 1 -- 172.21.15.115:0/2276887041 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa1c4097ed0 con 0x7fa1cc100290 2023-07-14T18:11:49.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.441+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa1cc1085b0 con 0x7fa1cc100290 2023-07-14T18:11:49.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.441+0000 7fa1d09c4700 1 -- 172.21.15.115:0/2276887041 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+2913 (secure 0 0 0) 0x7fa1c409b4f0 con 0x7fa1cc100290 2023-07-14T18:11:49.481 INFO:teuthology.orchestra.run.smithi115.stdout:e24 2023-07-14T18:11:49.481 INFO:teuthology.orchestra.run.smithi115.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:11:49.481 INFO:teuthology.orchestra.run.smithi115.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:11:49.481 INFO:teuthology.orchestra.run.smithi115.stdout:legacy client fscid: 1 2023-07-14T18:11:49.481 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:49.481 INFO:teuthology.orchestra.run.smithi115.stdout:Filesystem 'cephfs' (1) 2023-07-14T18:11:49.481 INFO:teuthology.orchestra.run.smithi115.stdout:fs_name cephfs 2023-07-14T18:11:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:epoch 24 2023-07-14T18:11:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-07-14T18:11:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:created 2023-07-14T18:10:17.634244+0000 2023-07-14T18:11:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:modified 2023-07-14T18:10:38.422616+0000 2023-07-14T18:11:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:tableserver 0 2023-07-14T18:11:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:root 0 2023-07-14T18:11:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:session_timeout 300 2023-07-14T18:11:49.482 INFO:teuthology.orchestra.run.smithi115.stdout:session_autoclose 300 2023-07-14T18:11:49.483 INFO:teuthology.orchestra.run.smithi115.stdout:max_file_size 1099511627776 2023-07-14T18:11:49.483 INFO:teuthology.orchestra.run.smithi115.stdout:max_xattr_size 65536 2023-07-14T18:11:49.483 INFO:teuthology.orchestra.run.smithi115.stdout:required_client_features {} 2023-07-14T18:11:49.483 INFO:teuthology.orchestra.run.smithi115.stdout:last_failure 0 2023-07-14T18:11:49.483 INFO:teuthology.orchestra.run.smithi115.stdout:last_failure_osd_epoch 0 2023-07-14T18:11:49.483 INFO:teuthology.orchestra.run.smithi115.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:11:49.483 INFO:teuthology.orchestra.run.smithi115.stdout:max_mds 3 2023-07-14T18:11:49.484 INFO:teuthology.orchestra.run.smithi115.stdout:in 0,1,2 2023-07-14T18:11:49.484 INFO:teuthology.orchestra.run.smithi115.stdout:up {0=24515,1=24493,2=24463} 2023-07-14T18:11:49.484 INFO:teuthology.orchestra.run.smithi115.stdout:failed 2023-07-14T18:11:49.484 INFO:teuthology.orchestra.run.smithi115.stdout:damaged 2023-07-14T18:11:49.484 INFO:teuthology.orchestra.run.smithi115.stdout:stopped 2023-07-14T18:11:49.484 INFO:teuthology.orchestra.run.smithi115.stdout:data_pools [3,4] 2023-07-14T18:11:49.484 INFO:teuthology.orchestra.run.smithi115.stdout:metadata_pool 2 2023-07-14T18:11:49.484 INFO:teuthology.orchestra.run.smithi115.stdout:inline_data disabled 2023-07-14T18:11:49.485 INFO:teuthology.orchestra.run.smithi115.stdout:balancer 2023-07-14T18:11:49.485 INFO:teuthology.orchestra.run.smithi115.stdout:bal_rank_mask -1 2023-07-14T18:11:49.485 INFO:teuthology.orchestra.run.smithi115.stdout:standby_count_wanted 0 2023-07-14T18:11:49.485 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.b{0:24515} state up:active seq 3 addr [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.485 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.h{0:24479} state up:standby-replay seq 1 addr [v2:172.21.15.150:6836/1462100932,v1:172.21.15.150:6837/1462100932] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.485 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.i{1:24493} state up:active seq 10 addr [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.485 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.c{1:24475} state up:standby-replay seq 1 addr [v2:172.21.15.187:6836/771796341,v1:172.21.15.187:6837/771796341] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.485 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.f{2:24463} state up:active seq 11 addr [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.486 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.e{2:24446} state up:standby-replay seq 1 addr [v2:172.21.15.150:6834/3630781789,v1:172.21.15.150:6835/3630781789] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.486 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:49.486 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:49.486 INFO:teuthology.orchestra.run.smithi115.stdout:Standby daemons: 2023-07-14T18:11:49.486 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:49.486 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.j{-1:14559} state up:standby seq 1 addr [v2:172.21.15.115:6834/3105890615,v1:172.21.15.115:6835/3105890615] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.486 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.d{-1:14565} state up:standby seq 1 addr [v2:172.21.15.115:6836/750049188,v1:172.21.15.115:6837/750049188] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.487 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.g{-1:14628} state up:standby seq 1 addr [v2:172.21.15.115:6838/2547274817,v1:172.21.15.115:6839/2547274817] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.487 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.a{-1:14640} state up:standby seq 1 addr [v2:172.21.15.115:6840/2429034282,v1:172.21.15.115:6841/2429034282] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.487 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.k{-1:24416} state up:standby seq 1 addr [v2:172.21.15.150:6832/1440309649,v1:172.21.15.150:6833/1440309649] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.487 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.l{-1:24427} state up:standby seq 1 addr [v2:172.21.15.187:6832/1471546028,v1:172.21.15.187:6833/1471546028] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:11:49.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa1b40a11e0 msgr2=0x7fa1b40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:49.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa1b40a11e0 0x7fa1b40a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc005950 tx=0x7fa1bc0058e0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc100290 msgr2=0x7fa1cc19cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:49.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc100290 0x7fa1cc19cf60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4006070 tx=0x7fa1c4004d30 comp rx=0 tx=0).stop 2023-07-14T18:11:49.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 shutdown_connections 2023-07-14T18:11:49.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa1b40a11e0 0x7fa1b40a3690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.488 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1cc100290 0x7fa1cc19cf60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1cc0fee60 0x7fa1cc1959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 --2- 172.21.15.115:0/2276887041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1cc0ff810 0x7fa1cc195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:49.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 >> 172.21.15.115:0/2276887041 conn(0x7fa1cc0f99c0 msgr2=0x7fa1cc0fbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:49.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 shutdown_connections 2023-07-14T18:11:49.489 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:49.443+0000 7fa1d19c6700 1 -- 172.21.15.115:0/2276887041 wait complete. 2023-07-14T18:11:49.489 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 24 2023-07-14T18:11:49.924 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- bash -c 'ceph osd dump' 2023-07-14T18:11:50.475 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:50.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:50 smithi115 ceph-mon[127451]: pgmap v299: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-07-14T18:11:50.607 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:50 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2276887041' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T18:11:50.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:50 smithi150 ceph-mon[128853]: pgmap v299: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-07-14T18:11:50.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:50 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2276887041' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T18:11:50.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:50 smithi187 ceph-mon[129317]: pgmap v299: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-07-14T18:11:50.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:50 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2276887041' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T18:11:51.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.765+0000 7ff0d3248700 1 -- 172.21.15.115:0/1770465223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 msgr2=0x7ff0cc0fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:51.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.765+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1770465223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 0x7ff0cc0fefc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009990 tx=0x7ff0c0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:51.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.765+0000 7ff0d3248700 1 -- 172.21.15.115:0/1770465223 shutdown_connections 2023-07-14T18:11:51.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.765+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1770465223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc100740 0x7ff0cc107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:51.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.765+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1770465223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 0x7ff0cc0fefc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:51.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.765+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1770465223 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0cc0ffdb0 0x7ff0cc100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:51.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.765+0000 7ff0d3248700 1 -- 172.21.15.115:0/1770465223 >> 172.21.15.115:0/1770465223 conn(0x7ff0cc0fa1e0 msgr2=0x7ff0cc0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:51.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.765+0000 7ff0d3248700 1 -- 172.21.15.115:0/1770465223 shutdown_connections 2023-07-14T18:11:51.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.765+0000 7ff0d3248700 1 -- 172.21.15.115:0/1770465223 wait complete. 2023-07-14T18:11:51.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d3248700 1 Processor -- start 2023-07-14T18:11:51.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d3248700 1 -- start start 2023-07-14T18:11:51.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d3248700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 0x7ff0cc197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:51.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d3248700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc0ffdb0 0x7ff0cc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:51.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d3248700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0cc100740 0x7ff0cc19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:51.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d3248700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc19f5f0 con 0x7ff0cc0febb0 2023-07-14T18:11:51.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d3248700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc19f730 con 0x7ff0cc0ffdb0 2023-07-14T18:11:51.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d3248700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc19f8c0 con 0x7ff0cc100740 2023-07-14T18:11:51.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d0fe4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 0x7ff0cc197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:51.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d0fe4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 0x7ff0cc197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53962/0 (socket says 172.21.15.115:53962) 2023-07-14T18:11:51.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d0fe4700 1 -- 172.21.15.115:0/1608048021 learned_addr learned my addr 172.21.15.115:0/1608048021 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:51.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0cbfff700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc0ffdb0 0x7ff0cc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:51.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d17e5700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0cc100740 0x7ff0cc19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:51.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d0fe4700 1 -- 172.21.15.115:0/1608048021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0cc100740 msgr2=0x7ff0cc19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:51.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d0fe4700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0cc100740 0x7ff0cc19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:51.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d0fe4700 1 -- 172.21.15.115:0/1608048021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc0ffdb0 msgr2=0x7ff0cc197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:51.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d0fe4700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc0ffdb0 0x7ff0cc197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:51.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.766+0000 7ff0d0fe4700 1 -- 172.21.15.115:0/1608048021 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c00095f0 con 0x7ff0cc0febb0 2023-07-14T18:11:51.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.767+0000 7ff0d0fe4700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 0x7ff0cc197a60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00098e0 tx=0x7ff0c0003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:51.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.767+0000 7ff0c9ffb700 1 -- 172.21.15.115:0/1608048021 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c001d070 con 0x7ff0cc0febb0 2023-07-14T18:11:51.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.767+0000 7ff0c9ffb700 1 -- 172.21.15.115:0/1608048021 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff0c000bd70 con 0x7ff0cc0febb0 2023-07-14T18:11:51.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.767+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0cc19fb40 con 0x7ff0cc0febb0 2023-07-14T18:11:51.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.767+0000 7ff0c9ffb700 1 -- 172.21.15.115:0/1608048021 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0022c60 con 0x7ff0cc0febb0 2023-07-14T18:11:51.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.767+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0cc19ff50 con 0x7ff0cc0febb0 2023-07-14T18:11:51.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.768+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0cc1061d0 con 0x7ff0cc0febb0 2023-07-14T18:11:51.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.769+0000 7ff0c9ffb700 1 -- 172.21.15.115:0/1608048021 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff0c0022470 con 0x7ff0cc0febb0 2023-07-14T18:11:51.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.770+0000 7ff0c9ffb700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff0bc0a11c0 0x7ff0bc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:51.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.770+0000 7ff0cbfff700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff0bc0a11c0 0x7ff0bc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:51.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.770+0000 7ff0c9ffb700 1 -- 172.21.15.115:0/1608048021 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff0c00ceb30 con 0x7ff0cc0febb0 2023-07-14T18:11:51.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.770+0000 7ff0cbfff700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff0bc0a11c0 0x7ff0bc0a3670 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800ac50 tx=0x7ff0b800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:51.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.773+0000 7ff0c9ffb700 1 -- 172.21.15.115:0/1608048021 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff0c0097ea0 con 0x7ff0cc0febb0 2023-07-14T18:11:52.003 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.964+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7ff0cc02cb30 con 0x7ff0cc0febb0 2023-07-14T18:11:52.004 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.965+0000 7ff0c9ffb700 1 -- 172.21.15.115:0/1608048021 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) v1 ==== 56+0+5632 (secure 0 0 0) 0x7ff0c009b4c0 con 0x7ff0cc0febb0 2023-07-14T18:11:52.004 INFO:teuthology.orchestra.run.smithi115.stdout:epoch 89 2023-07-14T18:11:52.005 INFO:teuthology.orchestra.run.smithi115.stdout:fsid a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:11:52.005 INFO:teuthology.orchestra.run.smithi115.stdout:created 2023-07-14T18:03:38.101205+0000 2023-07-14T18:11:52.005 INFO:teuthology.orchestra.run.smithi115.stdout:modified 2023-07-14T18:11:30.172466+0000 2023-07-14T18:11:52.005 INFO:teuthology.orchestra.run.smithi115.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-07-14T18:11:52.005 INFO:teuthology.orchestra.run.smithi115.stdout:crush_version 27 2023-07-14T18:11:52.005 INFO:teuthology.orchestra.run.smithi115.stdout:full_ratio 0.95 2023-07-14T18:11:52.005 INFO:teuthology.orchestra.run.smithi115.stdout:backfillfull_ratio 0.9 2023-07-14T18:11:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:nearfull_ratio 0.85 2023-07-14T18:11:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:require_min_compat_client luminous 2023-07-14T18:11:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:min_compat_client luminous 2023-07-14T18:11:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:require_osd_release reef 2023-07-14T18:11:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:stretch_mode_enabled false 2023-07-14T18:11:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-07-14T18:11:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-07-14T18:11:52.006 INFO:teuthology.orchestra.run.smithi115.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-07-14T18:11:52.007 INFO:teuthology.orchestra.run.smithi115.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 87 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-07-14T18:11:52.007 INFO:teuthology.orchestra.run.smithi115.stdout:max_osd 12 2023-07-14T18:11:52.007 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 up in weight 1 up_from 8 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6802/2677759926,v1:172.21.15.115:6803/2677759926] [v2:172.21.15.115:6804/2677759926,v1:172.21.15.115:6805/2677759926] exists,up 578188a6-96fa-444a-8a11-224bb5a7c098 2023-07-14T18:11:52.007 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6800/396669416,v1:172.21.15.150:6801/396669416] [v2:172.21.15.150:6802/396669416,v1:172.21.15.150:6803/396669416] exists,up ffda5af6-01b2-4a92-863f-7f6518223778 2023-07-14T18:11:52.007 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6800/262076123,v1:172.21.15.187:6801/262076123] [v2:172.21.15.187:6802/262076123,v1:172.21.15.187:6803/262076123] exists,up 83a08d79-e0ac-47ea-a061-5b92c4a33d3e 2023-07-14T18:11:52.007 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6810/3432098119,v1:172.21.15.115:6811/3432098119] [v2:172.21.15.115:6812/3432098119,v1:172.21.15.115:6813/3432098119] exists,up 9796f71f-7cff-4ee0-ae32-545fb5fc886d 2023-07-14T18:11:52.007 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 up in weight 1 up_from 32 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6808/3033896975,v1:172.21.15.150:6809/3033896975] [v2:172.21.15.150:6810/3033896975,v1:172.21.15.150:6811/3033896975] exists,up 82386140-1f12-43e9-9d70-6ca597009e77 2023-07-14T18:11:52.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 up in weight 1 up_from 37 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6808/3577781836,v1:172.21.15.187:6809/3577781836] [v2:172.21.15.187:6810/3577781836,v1:172.21.15.187:6811/3577781836] exists,up 9af1e805-b05f-422e-8f27-f95d74e9fec6 2023-07-14T18:11:52.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 up in weight 1 up_from 45 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6818/1235073533,v1:172.21.15.115:6819/1235073533] [v2:172.21.15.115:6820/1235073533,v1:172.21.15.115:6821/1235073533] exists,up 5bceb601-e8f0-440f-b674-3ce286cc4d3d 2023-07-14T18:11:52.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 up in weight 1 up_from 50 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6816/3128310736,v1:172.21.15.150:6817/3128310736] [v2:172.21.15.150:6818/3128310736,v1:172.21.15.150:6819/3128310736] exists,up bbe475b0-b4f9-41df-9615-a90c6fc62280 2023-07-14T18:11:52.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.8 up in weight 1 up_from 58 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6816/1513660091,v1:172.21.15.187:6817/1513660091] [v2:172.21.15.187:6818/1513660091,v1:172.21.15.187:6819/1513660091] exists,up 574a1fb8-fa71-4c03-812d-bb1d8da70f8b 2023-07-14T18:11:52.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.9 up in weight 1 up_from 66 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6826/3384303720,v1:172.21.15.115:6827/3384303720] [v2:172.21.15.115:6828/3384303720,v1:172.21.15.115:6829/3384303720] exists,up fd79ae74-eaa1-4b63-aad8-c77864ba2eed 2023-07-14T18:11:52.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.10 up in weight 1 up_from 71 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6824/4251241728,v1:172.21.15.150:6825/4251241728] [v2:172.21.15.150:6826/4251241728,v1:172.21.15.150:6827/4251241728] exists,up 2d66bdb3-b20d-4474-ab4b-9d0f81232272 2023-07-14T18:11:52.008 INFO:teuthology.orchestra.run.smithi115.stdout:osd.11 up in weight 1 up_from 78 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6824/2714221245,v1:172.21.15.187:6825/2714221245] [v2:172.21.15.187:6826/2714221245,v1:172.21.15.187:6827/2714221245] exists,up 646460b3-b30a-4570-8b1a-dce626e03f2c 2023-07-14T18:11:52.008 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 2.4 [2,11] 2023-07-14T18:11:52.009 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 2.5 [8,11] 2023-07-14T18:11:52.009 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 2.1a [8,1] 2023-07-14T18:11:52.009 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 2.1d [2,4] 2023-07-14T18:11:52.009 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 3.17 [4,8] 2023-07-14T18:11:52.009 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 3.2c [4,9] 2023-07-14T18:11:52.009 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 4.5 [5,8] 2023-07-14T18:11:52.009 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 4.e [9,11] 2023-07-14T18:11:52.009 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 4.10 [9,8] 2023-07-14T18:11:52.010 INFO:teuthology.orchestra.run.smithi115.stdout:pg_upmap_items 4.14 [9,10] 2023-07-14T18:11:52.010 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:6800/2169355495 expires 2023-07-15T18:04:28.808488+0000 2023-07-14T18:11:52.010 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:0/4031581902 expires 2023-07-15T18:04:28.808488+0000 2023-07-14T18:11:52.010 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:0/3311800921 expires 2023-07-15T18:04:28.808488+0000 2023-07-14T18:11:52.010 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:6801/2169355495 expires 2023-07-15T18:04:28.808488+0000 2023-07-14T18:11:52.010 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:0/2790161712 expires 2023-07-15T18:04:06.647341+0000 2023-07-14T18:11:52.010 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:0/2625519889 expires 2023-07-15T18:04:06.647341+0000 2023-07-14T18:11:52.010 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:6801/568655687 expires 2023-07-15T18:04:06.647341+0000 2023-07-14T18:11:52.011 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:0/3340901626 expires 2023-07-15T18:04:06.647341+0000 2023-07-14T18:11:52.011 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:6800/568655687 expires 2023-07-15T18:04:06.647341+0000 2023-07-14T18:11:52.011 INFO:teuthology.orchestra.run.smithi115.stdout:blocklist 172.21.15.115:0/2416566340 expires 2023-07-15T18:04:28.808488+0000 2023-07-14T18:11:52.011 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff0bc0a11c0 msgr2=0x7ff0bc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff0bc0a11c0 0x7ff0bc0a3670 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800ac50 tx=0x7ff0b800a380 comp rx=0 tx=0).stop 2023-07-14T18:11:52.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 msgr2=0x7ff0cc197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 0x7ff0cc197a60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00098e0 tx=0x7ff0c0003a50 comp rx=0 tx=0).stop 2023-07-14T18:11:52.012 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 shutdown_connections 2023-07-14T18:11:52.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff0bc0a11c0 0x7ff0bc0a3670 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff0cc100740 0x7ff0cc19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff0cc0febb0 0x7ff0cc197a60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 --2- 172.21.15.115:0/1608048021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc0ffdb0 0x7ff0cc197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 >> 172.21.15.115:0/1608048021 conn(0x7ff0cc0fa1e0 msgr2=0x7ff0cc103310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:52.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 shutdown_connections 2023-07-14T18:11:52.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:51.967+0000 7ff0d3248700 1 -- 172.21.15.115:0/1608048021 wait complete. 2023-07-14T18:11:52.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:52 smithi115 ceph-mon[127451]: pgmap v300: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-14T18:11:52.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:52 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1608048021' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T18:11:52.425 INFO:teuthology.run_tasks:Running task fs.ready... 2023-07-14T18:11:52.435 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:11:52.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.521+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/1006783803 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 msgr2=0x7f0ca0102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.521+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/1006783803 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 0x7f0ca0102a80 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c009990 tx=0x7f0c9c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.521+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/1006783803 shutdown_connections 2023-07-14T18:11:52.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.521+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/1006783803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ca01058e0 0x7f0ca0107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.521+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/1006783803 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 0x7f0ca0102a80 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.521+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/1006783803 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ca0102fc0 0x7f0ca01053a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.521+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/1006783803 >> 172.21.15.115:0/1006783803 conn(0x7f0ca00fa100 msgr2=0x7f0ca00fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:52.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.521+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/1006783803 shutdown_connections 2023-07-14T18:11:52.562 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.521+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/1006783803 wait complete. 2023-07-14T18:11:52.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca7bd1700 1 Processor -- start 2023-07-14T18:11:52.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca7bd1700 1 -- start start 2023-07-14T18:11:52.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca7bd1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 0x7f0ca0195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:52.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca7bd1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ca0102fc0 0x7f0ca0195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:52.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca7bd1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ca01058e0 0x7f0ca019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:52.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca7bd1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca019d3d0 con 0x7f0ca01006a0 2023-07-14T18:11:52.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca7bd1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca019d510 con 0x7f0ca01058e0 2023-07-14T18:11:52.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca7bd1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca019d6a0 con 0x7f0ca0102fc0 2023-07-14T18:11:52.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca596d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 0x7f0ca0195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:52.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca596d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 0x7f0ca0195850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53974/0 (socket says 172.21.15.115:53974) 2023-07-14T18:11:52.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca596d700 1 -- 172.21.15.115:0/147165452 learned_addr learned my addr 172.21.15.115:0/147165452 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:52.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca616e700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ca01058e0 0x7f0ca019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:52.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca596d700 1 -- 172.21.15.115:0/147165452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ca0102fc0 msgr2=0x7f0ca0195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca596d700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ca0102fc0 0x7f0ca0195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca596d700 1 -- 172.21.15.115:0/147165452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ca01058e0 msgr2=0x7f0ca019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca596d700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ca01058e0 0x7f0ca019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca596d700 1 -- 172.21.15.115:0/147165452 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c0095f0 con 0x7f0ca01006a0 2023-07-14T18:11:52.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.522+0000 7f0ca616e700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ca01058e0 0x7f0ca019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:52.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.523+0000 7f0ca596d700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 0x7f0ca0195850 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c006010 tx=0x7f0c9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:52.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.523+0000 7f0c96ffd700 1 -- 172.21.15.115:0/147165452 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c01d070 con 0x7f0ca01006a0 2023-07-14T18:11:52.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.523+0000 7f0c96ffd700 1 -- 172.21.15.115:0/147165452 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0c9c0041b0 con 0x7f0ca01006a0 2023-07-14T18:11:52.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.523+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca019d920 con 0x7f0ca01006a0 2023-07-14T18:11:52.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.523+0000 7f0c96ffd700 1 -- 172.21.15.115:0/147165452 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c022da0 con 0x7f0ca01006a0 2023-07-14T18:11:52.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.523+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca019dd40 con 0x7f0ca01006a0 2023-07-14T18:11:52.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.524+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca00fbc70 con 0x7f0ca01006a0 2023-07-14T18:11:52.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.525+0000 7f0c96ffd700 1 -- 172.21.15.115:0/147165452 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f0c9c022570 con 0x7f0ca01006a0 2023-07-14T18:11:52.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.525+0000 7f0c96ffd700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0c8c0a1220 0x7f0c8c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:52.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.525+0000 7f0ca516c700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0c8c0a1220 0x7f0c8c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:52.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.526+0000 7f0c96ffd700 1 -- 172.21.15.115:0/147165452 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0c9c0ce6f0 con 0x7f0ca01006a0 2023-07-14T18:11:52.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.526+0000 7f0ca516c700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0c8c0a1220 0x7f0c8c0a36d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0c90005950 tx=0x7f0c900058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:52.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.529+0000 7f0c96ffd700 1 -- 172.21.15.115:0/147165452 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0c9c097a60 con 0x7f0ca01006a0 2023-07-14T18:11:52.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:52 smithi150 ceph-mon[128853]: pgmap v300: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-14T18:11:52.622 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:52 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1608048021' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T18:11:52.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:52 smithi187 ceph-mon[129317]: pgmap v300: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-14T18:11:52.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:52 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1608048021' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T18:11:52.756 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.717+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ca002c8e0 con 0x7f0ca01006a0 2023-07-14T18:11:52.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.718+0000 7f0c96ffd700 1 -- 172.21.15.115:0/147165452 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24558 (secure 0 0 0) 0x7f0c9c09b080 con 0x7f0ca01006a0 2023-07-14T18:11:52.758 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:52.758 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":89,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:11:30.172466+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:11:52.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0c8c0a1220 msgr2=0x7f0c8c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0c8c0a1220 0x7f0c8c0a36d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0c90005950 tx=0x7f0c900058e0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.760 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 msgr2=0x7f0ca0195850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 0x7f0ca0195850 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c006010 tx=0x7f0c9c004730 comp rx=0 tx=0).stop 2023-07-14T18:11:52.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 shutdown_connections 2023-07-14T18:11:52.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f0c8c0a1220 0x7f0c8c0a36d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ca01058e0 0x7f0ca019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ca01006a0 0x7f0ca0195850 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 --2- 172.21.15.115:0/147165452 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0ca0102fc0 0x7f0ca0195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 >> 172.21.15.115:0/147165452 conn(0x7f0ca00fa100 msgr2=0x7f0ca0101300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:52.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 shutdown_connections 2023-07-14T18:11:52.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.721+0000 7f0ca7bd1700 1 -- 172.21.15.115:0/147165452 wait complete. 2023-07-14T18:11:52.773 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:10:15.471213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:10:16.791973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:10:20.663275+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:11:52.773 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:11:52.898 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.859+0000 7f71aed3a700 1 -- 172.21.15.115:0/2598170954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a8100770 msgr2=0x7f71a8107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.859+0000 7f71aed3a700 1 --2- 172.21.15.115:0/2598170954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a8100770 0x7f71a8107010 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f71a000b0a0 tx=0x7f71a000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.859+0000 7f71aed3a700 1 -- 172.21.15.115:0/2598170954 shutdown_connections 2023-07-14T18:11:52.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.859+0000 7f71aed3a700 1 --2- 172.21.15.115:0/2598170954 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a8100770 0x7f71a8107010 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.859+0000 7f71aed3a700 1 --2- 172.21.15.115:0/2598170954 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a80feb30 0x7f71a80fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.899 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.859+0000 7f71aed3a700 1 --2- 172.21.15.115:0/2598170954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a80ffdc0 0x7f71a8100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.859+0000 7f71aed3a700 1 -- 172.21.15.115:0/2598170954 >> 172.21.15.115:0/2598170954 conn(0x7f71a80fa150 msgr2=0x7f71a80fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:52.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.859+0000 7f71aed3a700 1 -- 172.21.15.115:0/2598170954 shutdown_connections 2023-07-14T18:11:52.900 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.859+0000 7f71aed3a700 1 -- 172.21.15.115:0/2598170954 wait complete. 2023-07-14T18:11:52.901 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71aed3a700 1 Processor -- start 2023-07-14T18:11:52.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71aed3a700 1 -- start start 2023-07-14T18:11:52.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71aed3a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a80feb30 0x7f71a8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:52.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71aed3a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a80ffdc0 0x7f71a8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:52.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71aed3a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a8100770 0x7f71a819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:52.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71aed3a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a819f680 con 0x7f71a80feb30 2023-07-14T18:11:52.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71aed3a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a819f7c0 con 0x7f71a80ffdc0 2023-07-14T18:11:52.902 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71aed3a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a819f950 con 0x7f71a8100770 2023-07-14T18:11:52.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71acad6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a80feb30 0x7f71a8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:52.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71acad6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a80feb30 0x7f71a8197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:53988/0 (socket says 172.21.15.115:53988) 2023-07-14T18:11:52.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71acad6700 1 -- 172.21.15.115:0/1032654077 learned_addr learned my addr 172.21.15.115:0/1032654077 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:52.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71ad2d7700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a8100770 0x7f71a819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:52.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71a7fff700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a80ffdc0 0x7f71a8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:52.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71acad6700 1 -- 172.21.15.115:0/1032654077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a8100770 msgr2=0x7f71a819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71acad6700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a8100770 0x7f71a819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.903 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71acad6700 1 -- 172.21.15.115:0/1032654077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a80ffdc0 msgr2=0x7f71a8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:52.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71acad6700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a80ffdc0 0x7f71a8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:52.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71acad6700 1 -- 172.21.15.115:0/1032654077 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a0009d00 con 0x7f71a80feb30 2023-07-14T18:11:52.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71ad2d7700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a8100770 0x7f71a819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:52.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.860+0000 7f71a7fff700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a80ffdc0 0x7f71a8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:52.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.861+0000 7f71acad6700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a80feb30 0x7f71a8197b00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f719c00c9b0 tx=0x7f719c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:52.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.861+0000 7f71a5ffb700 1 -- 172.21.15.115:0/1032654077 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c007b00 con 0x7f71a80feb30 2023-07-14T18:11:52.904 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.861+0000 7f71a5ffb700 1 -- 172.21.15.115:0/1032654077 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f719c007c60 con 0x7f71a80feb30 2023-07-14T18:11:52.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.861+0000 7f71a5ffb700 1 -- 172.21.15.115:0/1032654077 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c018660 con 0x7f71a80feb30 2023-07-14T18:11:52.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.861+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a819fc30 con 0x7f71a80feb30 2023-07-14T18:11:52.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.861+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a81a00f0 con 0x7f71a80feb30 2023-07-14T18:11:52.905 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.862+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a810a1b0 con 0x7f71a80feb30 2023-07-14T18:11:52.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.863+0000 7f71a5ffb700 1 -- 172.21.15.115:0/1032654077 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f719c00f450 con 0x7f71a80feb30 2023-07-14T18:11:52.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.864+0000 7f71a5ffb700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f71980a11e0 0x7f71980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:52.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.864+0000 7f71a7fff700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f71980a11e0 0x7f71980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:52.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.864+0000 7f71a5ffb700 1 -- 172.21.15.115:0/1032654077 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f719c0cced0 con 0x7f71a80feb30 2023-07-14T18:11:52.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.864+0000 7f71a7fff700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f71980a11e0 0x7f71980a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7194005950 tx=0x7f71940058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:52.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:52.868+0000 7f71a5ffb700 1 -- 172.21.15.115:0/1032654077 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f719c096150 con 0x7f71a80feb30 2023-07-14T18:11:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.059+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f71a81a0700 con 0x7f71a80feb30 2023-07-14T18:11:53.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.059+0000 7f71a5ffb700 1 -- 172.21.15.115:0/1032654077 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f719c099770 con 0x7f71a80feb30 2023-07-14T18:11:53.098 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:11:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f71980a11e0 msgr2=0x7f71980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f71980a11e0 0x7f71980a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7194005950 tx=0x7f71940058e0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a80feb30 msgr2=0x7f71a8197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.101 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a80feb30 0x7f71a8197b00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f719c00c9b0 tx=0x7f719c00cd70 comp rx=0 tx=0).stop 2023-07-14T18:11:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 shutdown_connections 2023-07-14T18:11:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f71980a11e0 0x7f71980a3690 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f71a8100770 0x7f71a819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f71a80feb30 0x7f71a8197b00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 --2- 172.21.15.115:0/1032654077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a80ffdc0 0x7f71a8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.061+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 >> 172.21.15.115:0/1032654077 conn(0x7f71a80fa150 msgr2=0x7f71a8068950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:53.102 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.062+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 shutdown_connections 2023-07-14T18:11:53.103 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.062+0000 7f71aed3a700 1 -- 172.21.15.115:0/1032654077 wait complete. 2023-07-14T18:11:53.113 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:11:53.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 -- 172.21.15.115:0/1264172117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40ffd00 msgr2=0x7fa6d4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/1264172117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40ffd00 0x7fa6d4100150 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009990 tx=0x7fa6c4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.237 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 -- 172.21.15.115:0/1264172117 shutdown_connections 2023-07-14T18:11:53.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/1264172117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6d4100690 0x7fa6d4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/1264172117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d40feb10 0x7fa6d40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/1264172117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40ffd00 0x7fa6d4100150 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 -- 172.21.15.115:0/1264172117 >> 172.21.15.115:0/1264172117 conn(0x7fa6d40fa130 msgr2=0x7fa6d40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:53.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 -- 172.21.15.115:0/1264172117 shutdown_connections 2023-07-14T18:11:53.238 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 -- 172.21.15.115:0/1264172117 wait complete. 2023-07-14T18:11:53.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 Processor -- start 2023-07-14T18:11:53.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 -- start start 2023-07-14T18:11:53.240 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40feb10 0x7fa6d4068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.197+0000 7fa6db3c9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6d40ffd00 0x7fa6d419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6db3c9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d4100690 0x7fa6d419cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6db3c9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d419d460 con 0x7fa6d40feb10 2023-07-14T18:11:53.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6db3c9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d419d5d0 con 0x7fa6d4100690 2023-07-14T18:11:53.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6db3c9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d419d790 con 0x7fa6d40ffd00 2023-07-14T18:11:53.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9165700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40feb10 0x7fa6d4068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.241 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9165700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40feb10 0x7fa6d4068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54002/0 (socket says 172.21.15.115:54002) 2023-07-14T18:11:53.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9165700 1 -- 172.21.15.115:0/4290104610 learned_addr learned my addr 172.21.15.115:0/4290104610 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:53.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d8964700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6d40ffd00 0x7fa6d419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9966700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d4100690 0x7fa6d419cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9165700 1 -- 172.21.15.115:0/4290104610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6d40ffd00 msgr2=0x7fa6d419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9165700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6d40ffd00 0x7fa6d419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.242 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9165700 1 -- 172.21.15.115:0/4290104610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d4100690 msgr2=0x7fa6d419cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9165700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d4100690 0x7fa6d419cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9165700 1 -- 172.21.15.115:0/4290104610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c40095f0 con 0x7fa6d40feb10 2023-07-14T18:11:53.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6d9165700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40feb10 0x7fa6d4068330 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000d980 tx=0x7fa6d000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:53.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6ca7fc700 1 -- 172.21.15.115:0/4290104610 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d000de60 con 0x7fa6d40feb10 2023-07-14T18:11:53.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6ca7fc700 1 -- 172.21.15.115:0/4290104610 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa6d00044b0 con 0x7fa6d40feb10 2023-07-14T18:11:53.243 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d419da70 con 0x7fa6d40feb10 2023-07-14T18:11:53.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6ca7fc700 1 -- 172.21.15.115:0/4290104610 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0003b40 con 0x7fa6d40feb10 2023-07-14T18:11:53.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.198+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d419df80 con 0x7fa6d40feb10 2023-07-14T18:11:53.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.199+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d4105b80 con 0x7fa6d40feb10 2023-07-14T18:11:53.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.200+0000 7fa6ca7fc700 1 -- 172.21.15.115:0/4290104610 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fa6d0004870 con 0x7fa6d40feb10 2023-07-14T18:11:53.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.201+0000 7fa6ca7fc700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa6c00a11e0 0x7fa6c00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.201+0000 7fa6d8964700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa6c00a11e0 0x7fa6c00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.201+0000 7fa6ca7fc700 1 -- 172.21.15.115:0/4290104610 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa6d00ccec0 con 0x7fa6d40feb10 2023-07-14T18:11:53.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.201+0000 7fa6d8964700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa6c00a11e0 0x7fa6c00a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400b5c0 tx=0x7fa6c4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:53.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.204+0000 7fa6ca7fc700 1 -- 172.21.15.115:0/4290104610 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa6d0096140 con 0x7fa6d40feb10 2023-07-14T18:11:53.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.394+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa6d402ca70 con 0x7fa6d40feb10 2023-07-14T18:11:53.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.394+0000 7fa6ca7fc700 1 -- 172.21.15.115:0/4290104610 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa6d0009940 con 0x7fa6d40feb10 2023-07-14T18:11:53.435 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:11:53.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa6c00a11e0 msgr2=0x7fa6c00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa6c00a11e0 0x7fa6c00a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400b5c0 tx=0x7fa6c4005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40feb10 msgr2=0x7fa6d4068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40feb10 0x7fa6d4068330 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000d980 tx=0x7fa6d000dd40 comp rx=0 tx=0).stop 2023-07-14T18:11:53.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 shutdown_connections 2023-07-14T18:11:53.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fa6c00a11e0 0x7fa6c00a3690 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6d4100690 0x7fa6d419cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa6d40feb10 0x7fa6d4068330 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 --2- 172.21.15.115:0/4290104610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6d40ffd00 0x7fa6d419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.396+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 >> 172.21.15.115:0/4290104610 conn(0x7fa6d40fa130 msgr2=0x7fa6d40fc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:53.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.397+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 shutdown_connections 2023-07-14T18:11:53.438 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.397+0000 7fa6db3c9700 1 -- 172.21.15.115:0/4290104610 wait complete. 2023-07-14T18:11:53.449 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:11:53.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.533+0000 7fda7008f700 1 -- 172.21.15.115:0/195702628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda681006e0 msgr2=0x7fda68106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.533+0000 7fda7008f700 1 --2- 172.21.15.115:0/195702628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda681006e0 0x7fda68106f80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fda6000b0a0 tx=0x7fda6000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.533+0000 7fda7008f700 1 -- 172.21.15.115:0/195702628 shutdown_connections 2023-07-14T18:11:53.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.533+0000 7fda7008f700 1 --2- 172.21.15.115:0/195702628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda681006e0 0x7fda68106f80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.533+0000 7fda7008f700 1 --2- 172.21.15.115:0/195702628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda680ff020 0x7fda680ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.533+0000 7fda7008f700 1 --2- 172.21.15.115:0/195702628 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda680ffd30 0x7fda681001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.533+0000 7fda7008f700 1 -- 172.21.15.115:0/195702628 >> 172.21.15.115:0/195702628 conn(0x7fda680f98a0 msgr2=0x7fda680fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:53.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 -- 172.21.15.115:0/195702628 shutdown_connections 2023-07-14T18:11:53.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 -- 172.21.15.115:0/195702628 wait complete. 2023-07-14T18:11:53.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 Processor -- start 2023-07-14T18:11:53.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 -- start start 2023-07-14T18:11:53.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda680ff020 0x7fda68102ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda680ffd30 0x7fda68101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda681006e0 0x7fda68101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda68101df0 con 0x7fda680ffd30 2023-07-14T18:11:53.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda68101f60 con 0x7fda680ff020 2023-07-14T18:11:53.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda7008f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6819fa00 con 0x7fda681006e0 2023-07-14T18:11:53.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda6d62a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda680ffd30 0x7fda68101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda6d62a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda680ffd30 0x7fda68101110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54018/0 (socket says 172.21.15.115:54018) 2023-07-14T18:11:53.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.534+0000 7fda6d62a700 1 -- 172.21.15.115:0/3663349887 learned_addr learned my addr 172.21.15.115:0/3663349887 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:53.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda6de2b700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda680ff020 0x7fda68102ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda6e62c700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda681006e0 0x7fda68101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda6d62a700 1 -- 172.21.15.115:0/3663349887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda681006e0 msgr2=0x7fda68101650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda6d62a700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda681006e0 0x7fda68101650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda6d62a700 1 -- 172.21.15.115:0/3663349887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda680ff020 msgr2=0x7fda68102ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda6d62a700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda680ff020 0x7fda68102ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda6d62a700 1 -- 172.21.15.115:0/3663349887 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda60009d00 con 0x7fda680ffd30 2023-07-14T18:11:53.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda6d62a700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda680ffd30 0x7fda68101110 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fda5800e970 tx=0x7fda5800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:53.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda5effd700 1 -- 172.21.15.115:0/3663349887 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda58009790 con 0x7fda680ffd30 2023-07-14T18:11:53.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda5effd700 1 -- 172.21.15.115:0/3663349887 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fda5800c7e0 con 0x7fda680ffd30 2023-07-14T18:11:53.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6819fce0 con 0x7fda680ffd30 2023-07-14T18:11:53.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda5effd700 1 -- 172.21.15.115:0/3663349887 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda58003bd0 con 0x7fda680ffd30 2023-07-14T18:11:53.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.535+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda681a01a0 con 0x7fda680ffd30 2023-07-14T18:11:53.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.536+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6810a140 con 0x7fda680ffd30 2023-07-14T18:11:53.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.537+0000 7fda5effd700 1 -- 172.21.15.115:0/3663349887 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fda5800c950 con 0x7fda680ffd30 2023-07-14T18:11:53.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.538+0000 7fda5effd700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda540a11e0 0x7fda540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.581 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.538+0000 7fda6de2b700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda540a11e0 0x7fda540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.538+0000 7fda5effd700 1 -- 172.21.15.115:0/3663349887 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fda58014070 con 0x7fda680ffd30 2023-07-14T18:11:53.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.538+0000 7fda6de2b700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda540a11e0 0x7fda540a3690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fda68067310 tx=0x7fda64009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:53.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.541+0000 7fda5effd700 1 -- 172.21.15.115:0/3663349887 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fda580963c0 con 0x7fda680ffd30 2023-07-14T18:11:53.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.728+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fda681a0810 con 0x7fda680ffd30 2023-07-14T18:11:53.769 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.729+0000 7fda5effd700 1 -- 172.21.15.115:0/3663349887 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fda580999e0 con 0x7fda680ffd30 2023-07-14T18:11:53.769 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 64 2023-07-14T18:11:53.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.730+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda540a11e0 msgr2=0x7fda540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.730+0000 7fda7008f700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda540a11e0 0x7fda540a3690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fda68067310 tx=0x7fda64009260 comp rx=0 tx=0).stop 2023-07-14T18:11:53.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.730+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda680ffd30 msgr2=0x7fda68101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.730+0000 7fda7008f700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda680ffd30 0x7fda68101110 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fda5800e970 tx=0x7fda5800ec80 comp rx=0 tx=0).stop 2023-07-14T18:11:53.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.731+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 shutdown_connections 2023-07-14T18:11:53.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.731+0000 7fda7008f700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fda540a11e0 0x7fda540a3690 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.731+0000 7fda7008f700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fda681006e0 0x7fda68101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.731+0000 7fda7008f700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda680ff020 0x7fda68102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.731+0000 7fda7008f700 1 --2- 172.21.15.115:0/3663349887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fda680ffd30 0x7fda68101110 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.731+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 >> 172.21.15.115:0/3663349887 conn(0x7fda680f98a0 msgr2=0x7fda680fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:53.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.731+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 shutdown_connections 2023-07-14T18:11:53.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.731+0000 7fda7008f700 1 -- 172.21.15.115:0/3663349887 wait complete. 2023-07-14T18:11:53.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:53 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/147165452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:53.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:53 smithi115 ceph-mon[127451]: pgmap v301: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-14T18:11:53.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:53 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1032654077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:53.782 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:53 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4290104610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:53.783 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:11:53.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:53 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/147165452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:53.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:53 smithi150 ceph-mon[128853]: pgmap v301: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-14T18:11:53.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:53 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1032654077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:53.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:53 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/4290104610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:53.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 -- 172.21.15.115:0/889076124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ff060 msgr2=0x7eff180ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/889076124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ff060 0x7eff180ff470 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7eff08009990 tx=0x7eff08009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 -- 172.21.15.115:0/889076124 shutdown_connections 2023-07-14T18:11:53.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/889076124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff18100720 0x7eff18106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/889076124 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff180ff060 0x7eff180ff470 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/889076124 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff180ffd70 0x7eff181001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 -- 172.21.15.115:0/889076124 >> 172.21.15.115:0/889076124 conn(0x7eff180f98e0 msgr2=0x7eff180fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:53.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 -- 172.21.15.115:0/889076124 shutdown_connections 2023-07-14T18:11:53.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 -- 172.21.15.115:0/889076124 wait complete. 2023-07-14T18:11:53.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 Processor -- start 2023-07-14T18:11:53.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.867+0000 7eff1e5d3700 1 -- start start 2023-07-14T18:11:53.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1e5d3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff180ff060 0x7eff180fe740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1e5d3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff180ffd70 0x7eff1819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1e5d3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff18100720 0x7eff1819cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1e5d3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1819d440 con 0x7eff18100720 2023-07-14T18:11:53.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1e5d3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1819d5b0 con 0x7eff180ff060 2023-07-14T18:11:53.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1e5d3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1819d770 con 0x7eff180ffd70 2023-07-14T18:11:53.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1cb70700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff18100720 0x7eff1819cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1cb70700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff18100720 0x7eff1819cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54034/0 (socket says 172.21.15.115:54034) 2023-07-14T18:11:53.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1cb70700 1 -- 172.21.15.115:0/1330840018 learned_addr learned my addr 172.21.15.115:0/1330840018 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:53.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff17fff700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff180ff060 0x7eff180fe740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff177fe700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff180ffd70 0x7eff1819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1cb70700 1 -- 172.21.15.115:0/1330840018 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff180ffd70 msgr2=0x7eff1819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1cb70700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff180ffd70 0x7eff1819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1cb70700 1 -- 172.21.15.115:0/1330840018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff180ff060 msgr2=0x7eff180fe740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:53.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1cb70700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff180ff060 0x7eff180fe740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:53.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1cb70700 1 -- 172.21.15.115:0/1330840018 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff080095f0 con 0x7eff18100720 2023-07-14T18:11:53.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff177fe700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff180ffd70 0x7eff1819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:53.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1cb70700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff18100720 0x7eff1819cc70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7eff1000ada0 tx=0x7eff1000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:53.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff157fa700 1 -- 172.21.15.115:0/1330840018 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1000f070 con 0x7eff18100720 2023-07-14T18:11:53.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1819da50 con 0x7eff18100720 2023-07-14T18:11:53.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff157fa700 1 -- 172.21.15.115:0/1330840018 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7eff10007b30 con 0x7eff18100720 2023-07-14T18:11:53.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff157fa700 1 -- 172.21.15.115:0/1330840018 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff100053e0 con 0x7eff18100720 2023-07-14T18:11:53.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.868+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1819df10 con 0x7eff18100720 2023-07-14T18:11:53.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.869+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1810a220 con 0x7eff18100720 2023-07-14T18:11:53.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.870+0000 7eff157fa700 1 -- 172.21.15.115:0/1330840018 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7eff1000c980 con 0x7eff18100720 2023-07-14T18:11:53.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.871+0000 7eff157fa700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7eff040a11e0 0x7eff040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:53.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.871+0000 7eff17fff700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7eff040a11e0 0x7eff040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:53.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.871+0000 7eff157fa700 1 -- 172.21.15.115:0/1330840018 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7eff100cc520 con 0x7eff18100720 2023-07-14T18:11:53.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.871+0000 7eff17fff700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7eff040a11e0 0x7eff040a3690 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7eff0800b5c0 tx=0x7eff08005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:53.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:53.875+0000 7eff157fa700 1 -- 172.21.15.115:0/1330840018 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7eff10095890 con 0x7eff18100720 2023-07-14T18:11:53.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:53 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/147165452' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:53.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:53 smithi187 ceph-mon[129317]: pgmap v301: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-14T18:11:53.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:53 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1032654077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:53.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:53 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/4290104610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:54.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.065+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7eff18105b20 con 0x7eff18100720 2023-07-14T18:11:54.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.066+0000 7eff157fa700 1 -- 172.21.15.115:0/1330840018 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7eff1000cc30 con 0x7eff18100720 2023-07-14T18:11:54.106 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:11:54.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7eff040a11e0 msgr2=0x7eff040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7eff040a11e0 0x7eff040a3690 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7eff0800b5c0 tx=0x7eff08005fb0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff18100720 msgr2=0x7eff1819cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff18100720 0x7eff1819cc70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7eff1000ada0 tx=0x7eff1000c330 comp rx=0 tx=0).stop 2023-07-14T18:11:54.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 shutdown_connections 2023-07-14T18:11:54.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7eff040a11e0 0x7eff040a3690 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7eff18100720 0x7eff1819cc70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff180ff060 0x7eff180fe740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 --2- 172.21.15.115:0/1330840018 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff180ffd70 0x7eff1819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 >> 172.21.15.115:0/1330840018 conn(0x7eff180f98e0 msgr2=0x7eff180fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:54.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 shutdown_connections 2023-07-14T18:11:54.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.068+0000 7eff1e5d3700 1 -- 172.21.15.115:0/1330840018 wait complete. 2023-07-14T18:11:54.120 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:11:54.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.203+0000 7fdfed1a3700 1 -- 172.21.15.115:0/1069659969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80ffed0 msgr2=0x7fdfe8106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.203+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/1069659969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80ffed0 0x7fdfe8106790 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000b0a0 tx=0x7fdfe000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.244 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.203+0000 7fdfed1a3700 1 -- 172.21.15.115:0/1069659969 shutdown_connections 2023-07-14T18:11:54.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.203+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/1069659969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80ffed0 0x7fdfe8106790 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.203+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/1069659969 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfe80fe350 0x7fdfe80fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.203+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/1069659969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfe80ff540 0x7fdfe80ff990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.203+0000 7fdfed1a3700 1 -- 172.21.15.115:0/1069659969 >> 172.21.15.115:0/1069659969 conn(0x7fdfe80f9990 msgr2=0x7fdfe80fbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:54.245 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.203+0000 7fdfed1a3700 1 -- 172.21.15.115:0/1069659969 shutdown_connections 2023-07-14T18:11:54.246 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.204+0000 7fdfed1a3700 1 -- 172.21.15.115:0/1069659969 wait complete. 2023-07-14T18:11:54.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.204+0000 7fdfed1a3700 1 Processor -- start 2023-07-14T18:11:54.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfed1a3700 1 -- start start 2023-07-14T18:11:54.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfed1a3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80fe350 0x7fdfe8194e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfed1a3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfe80ff540 0x7fdfe81953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfed1a3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfe80ffed0 0x7fdfe819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfed1a3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe819c380 con 0x7fdfe80fe350 2023-07-14T18:11:54.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfed1a3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe819c4f0 con 0x7fdfe80ffed0 2023-07-14T18:11:54.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfed1a3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe819c6b0 con 0x7fdfe80ff540 2023-07-14T18:11:54.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe6d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80fe350 0x7fdfe8194e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe6d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80fe350 0x7fdfe8194e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54044/0 (socket says 172.21.15.115:54044) 2023-07-14T18:11:54.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe6d9d700 1 -- 172.21.15.115:0/2156424937 learned_addr learned my addr 172.21.15.115:0/2156424937 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:54.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe659c700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfe80ff540 0x7fdfe81953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.249 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe759e700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfe80ffed0 0x7fdfe819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe6d9d700 1 -- 172.21.15.115:0/2156424937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfe80ff540 msgr2=0x7fdfe81953a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe6d9d700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfe80ff540 0x7fdfe81953a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe6d9d700 1 -- 172.21.15.115:0/2156424937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfe80ffed0 msgr2=0x7fdfe819bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe6d9d700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfe80ffed0 0x7fdfe819bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe6d9d700 1 -- 172.21.15.115:0/2156424937 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe0009d00 con 0x7fdfe80fe350 2023-07-14T18:11:54.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe759e700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfe80ffed0 0x7fdfe819bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:54.250 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.205+0000 7fdfe6d9d700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80fe350 0x7fdfe8194e60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800e970 tx=0x7fdfd800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:54.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.206+0000 7fdfcffff700 1 -- 172.21.15.115:0/2156424937 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8009790 con 0x7fdfe80fe350 2023-07-14T18:11:54.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.206+0000 7fdfcffff700 1 -- 172.21.15.115:0/2156424937 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdfd800eec0 con 0x7fdfe80fe350 2023-07-14T18:11:54.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.206+0000 7fdfcffff700 1 -- 172.21.15.115:0/2156424937 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8003c10 con 0x7fdfe80fe350 2023-07-14T18:11:54.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.206+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe819c990 con 0x7fdfe80fe350 2023-07-14T18:11:54.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.206+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe819cea0 con 0x7fdfe80fe350 2023-07-14T18:11:54.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.207+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe81058b0 con 0x7fdfe80fe350 2023-07-14T18:11:54.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.208+0000 7fdfcffff700 1 -- 172.21.15.115:0/2156424937 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fdfd8003db0 con 0x7fdfe80fe350 2023-07-14T18:11:54.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.208+0000 7fdfcffff700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfd00a12a0 0x7fdfd00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.208+0000 7fdfe659c700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfd00a12a0 0x7fdfd00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.209+0000 7fdfcffff700 1 -- 172.21.15.115:0/2156424937 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdfd8014070 con 0x7fdfe80fe350 2023-07-14T18:11:54.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.209+0000 7fdfe659c700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfd00a12a0 0x7fdfd00a3750 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8068700 tx=0x7fdfdc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:54.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.213+0000 7fdfcffff700 1 -- 172.21.15.115:0/2156424937 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdfd8096290 con 0x7fdfe80fe350 2023-07-14T18:11:54.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.440+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfe802c8e0 con 0x7fdfe80fe350 2023-07-14T18:11:54.483 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.443+0000 7fdfcffff700 1 -- 172.21.15.115:0/2156424937 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10851 (secure 0 0 0) 0x7fdfd80998b0 con 0x7fdfe80fe350 2023-07-14T18:11:54.484 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:54.484 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:10:38.422616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24493,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24446":{"gid":24446,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:11:54.485 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfd00a12a0 msgr2=0x7fdfd00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfd00a12a0 0x7fdfd00a3750 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8068700 tx=0x7fdfdc009260 comp rx=0 tx=0).stop 2023-07-14T18:11:54.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80fe350 msgr2=0x7fdfe8194e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80fe350 0x7fdfe8194e60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdfd800e970 tx=0x7fdfd800ec80 comp rx=0 tx=0).stop 2023-07-14T18:11:54.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 shutdown_connections 2023-07-14T18:11:54.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdfd00a12a0 0x7fdfd00a3750 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.486 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfe80ffed0 0x7fdfe819bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdfe80fe350 0x7fdfe8194e60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 --2- 172.21.15.115:0/2156424937 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfe80ff540 0x7fdfe81953a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 >> 172.21.15.115:0/2156424937 conn(0x7fdfe80f9990 msgr2=0x7fdfe81029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:54.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 shutdown_connections 2023-07-14T18:11:54.487 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.445+0000 7fdfed1a3700 1 -- 172.21.15.115:0/2156424937 wait complete. 2023-07-14T18:11:54.488 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 24 2023-07-14T18:11:54.498 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:11:54.619 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.579+0000 7fbce456f700 1 -- 172.21.15.115:0/2363890682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 msgr2=0x7fbcdc0fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.579+0000 7fbce456f700 1 --2- 172.21.15.115:0/2363890682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 0x7fbcdc0fe690 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009990 tx=0x7fbccc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.579+0000 7fbce456f700 1 -- 172.21.15.115:0/2363890682 shutdown_connections 2023-07-14T18:11:54.620 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.579+0000 7fbce456f700 1 --2- 172.21.15.115:0/2363890682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0ffe00 0x7fbcdc1066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.579+0000 7fbce456f700 1 --2- 172.21.15.115:0/2363890682 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 0x7fbcdc0fe690 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.579+0000 7fbce456f700 1 --2- 172.21.15.115:0/2363890682 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcdc0ff470 0x7fbcdc0ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.579+0000 7fbce456f700 1 -- 172.21.15.115:0/2363890682 >> 172.21.15.115:0/2363890682 conn(0x7fbcdc0f98a0 msgr2=0x7fbcdc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:54.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.579+0000 7fbce456f700 1 -- 172.21.15.115:0/2363890682 shutdown_connections 2023-07-14T18:11:54.621 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.579+0000 7fbce456f700 1 -- 172.21.15.115:0/2363890682 wait complete. 2023-07-14T18:11:54.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce456f700 1 Processor -- start 2023-07-14T18:11:54.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce456f700 1 -- start start 2023-07-14T18:11:54.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce456f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 0x7fbcdc195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce456f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0ff470 0x7fbcdc195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce456f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcdc0ffe00 0x7fbcdc19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce456f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19d3f0 con 0x7fbcdc0fe280 2023-07-14T18:11:54.624 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce456f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19d530 con 0x7fbcdc0ff470 2023-07-14T18:11:54.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce456f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19d6c0 con 0x7fbcdc0ffe00 2023-07-14T18:11:54.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce230b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 0x7fbcdc195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce230b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 0x7fbcdc195870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54056/0 (socket says 172.21.15.115:54056) 2023-07-14T18:11:54.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce230b700 1 -- 172.21.15.115:0/3973690026 learned_addr learned my addr 172.21.15.115:0/3973690026 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:54.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce1b0a700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0ff470 0x7fbcdc195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce2b0c700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcdc0ffe00 0x7fbcdc19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce230b700 1 -- 172.21.15.115:0/3973690026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcdc0ffe00 msgr2=0x7fbcdc19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.625 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce230b700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcdc0ffe00 0x7fbcdc19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce230b700 1 -- 172.21.15.115:0/3973690026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0ff470 msgr2=0x7fbcdc195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce230b700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0ff470 0x7fbcdc195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce230b700 1 -- 172.21.15.115:0/3973690026 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc0095f0 con 0x7fbcdc0fe280 2023-07-14T18:11:54.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.580+0000 7fbce1b0a700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0ff470 0x7fbcdc195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:54.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.581+0000 7fbce230b700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 0x7fbcdc195870 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbccc004730 tx=0x7fbccc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:54.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.581+0000 7fbcd37fe700 1 -- 172.21.15.115:0/3973690026 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc01d070 con 0x7fbcdc0fe280 2023-07-14T18:11:54.626 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.581+0000 7fbcd37fe700 1 -- 172.21.15.115:0/3973690026 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbccc004080 con 0x7fbcdc0fe280 2023-07-14T18:11:54.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.581+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcdc19d940 con 0x7fbcdc0fe280 2023-07-14T18:11:54.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.581+0000 7fbcd37fe700 1 -- 172.21.15.115:0/3973690026 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc004ea0 con 0x7fbcdc0fe280 2023-07-14T18:11:54.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.581+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcdc19dd60 con 0x7fbcdc0fe280 2023-07-14T18:11:54.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.582+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcdc109150 con 0x7fbcdc0fe280 2023-07-14T18:11:54.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.583+0000 7fbcd37fe700 1 -- 172.21.15.115:0/3973690026 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fbccc003bf0 con 0x7fbcdc0fe280 2023-07-14T18:11:54.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.583+0000 7fbcd37fe700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcc80a1220 0x7fbcc80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.627 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.583+0000 7fbce1b0a700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcc80a1220 0x7fbcc80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.583+0000 7fbcd37fe700 1 -- 172.21.15.115:0/3973690026 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbccc0cddf0 con 0x7fbcdc0fe280 2023-07-14T18:11:54.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.584+0000 7fbce1b0a700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcc80a1220 0x7fbcc80a36d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8005950 tx=0x7fbcd80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:54.628 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.587+0000 7fbcd37fe700 1 -- 172.21.15.115:0/3973690026 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbccc097160 con 0x7fbcdc0fe280 2023-07-14T18:11:54.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.776+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcdc109340 con 0x7fbcdc0fe280 2023-07-14T18:11:54.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.778+0000 7fbcd37fe700 1 -- 172.21.15.115:0/3973690026 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24558 (secure 0 0 0) 0x7fbccc09a780 con 0x7fbcdc0fe280 2023-07-14T18:11:54.818 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:54.819 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":89,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:11:30.172466+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:11:54.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcc80a1220 msgr2=0x7fbcc80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcc80a1220 0x7fbcc80a36d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8005950 tx=0x7fbcd80058e0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 msgr2=0x7fbcdc195870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 0x7fbcdc195870 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbccc004730 tx=0x7fbccc009580 comp rx=0 tx=0).stop 2023-07-14T18:11:54.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 shutdown_connections 2023-07-14T18:11:54.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fbcc80a1220 0x7fbcc80a36d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcdc0ffe00 0x7fbcdc19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbcdc0fe280 0x7fbcdc195870 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 --2- 172.21.15.115:0/3973690026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcdc0ff470 0x7fbcdc195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 >> 172.21.15.115:0/3973690026 conn(0x7fbcdc0f98a0 msgr2=0x7fbcdc104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:54.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 shutdown_connections 2023-07-14T18:11:54.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.780+0000 7fbce456f700 1 -- 172.21.15.115:0/3973690026 wait complete. 2023-07-14T18:11:54.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:54 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3663349887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:54.831 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:54 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1330840018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:54.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:10:15.471213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:10:16.791973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:10:20.663275+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:11:54.833 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:11:54.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:54 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3663349887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:54.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:54 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1330840018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:54.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:54 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3663349887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:54.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:54 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1330840018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:54.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 -- 172.21.15.115:0/2611389249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f4100700 msgr2=0x7f63f4106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/2611389249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f4100700 0x7f63f4106fc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00b0a0 tx=0x7f63ec00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 -- 172.21.15.115:0/2611389249 shutdown_connections 2023-07-14T18:11:54.961 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/2611389249 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f4100700 0x7f63f4106fc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/2611389249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f40feb80 0x7f63f40fef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/2611389249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f40ffd70 0x7f63f41001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 -- 172.21.15.115:0/2611389249 >> 172.21.15.115:0/2611389249 conn(0x7f63f40fa1a0 msgr2=0x7f63f40fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:54.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 -- 172.21.15.115:0/2611389249 shutdown_connections 2023-07-14T18:11:54.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 -- 172.21.15.115:0/2611389249 wait complete. 2023-07-14T18:11:54.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 Processor -- start 2023-07-14T18:11:54.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 -- start start 2023-07-14T18:11:54.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f40feb80 0x7f63f4194870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f40ffd70 0x7f63f4194db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f4100700 0x7f63f419be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f419c3f0 con 0x7f63f40feb80 2023-07-14T18:11:54.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f419c530 con 0x7f63f4100700 2023-07-14T18:11:54.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.918+0000 7f63fb0a4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f419c6c0 con 0x7f63f40ffd70 2023-07-14T18:11:54.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f8e40700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f40feb80 0x7f63f4194870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f8e40700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f40feb80 0x7f63f4194870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54070/0 (socket says 172.21.15.115:54070) 2023-07-14T18:11:54.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f8e40700 1 -- 172.21.15.115:0/1193211307 learned_addr learned my addr 172.21.15.115:0/1193211307 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:54.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f3fff700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f40ffd70 0x7f63f4194db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f9641700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f4100700 0x7f63f419be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f8e40700 1 -- 172.21.15.115:0/1193211307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f40ffd70 msgr2=0x7f63f4194db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f8e40700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f40ffd70 0x7f63f4194db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f8e40700 1 -- 172.21.15.115:0/1193211307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f4100700 msgr2=0x7f63f419be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:54.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f8e40700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f4100700 0x7f63f419be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:54.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f8e40700 1 -- 172.21.15.115:0/1193211307 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ec009d00 con 0x7f63f40feb80 2023-07-14T18:11:54.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f9641700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f4100700 0x7f63f419be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:54.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f8e40700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f40feb80 0x7f63f4194870 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c970 tx=0x7f63e800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:54.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f1ffb700 1 -- 172.21.15.115:0/1193211307 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8004220 con 0x7f63f40feb80 2023-07-14T18:11:54.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f1ffb700 1 -- 172.21.15.115:0/1193211307 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f63e8007d30 con 0x7f63f40feb80 2023-07-14T18:11:54.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f419c9a0 con 0x7f63f40feb80 2023-07-14T18:11:54.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63f1ffb700 1 -- 172.21.15.115:0/1193211307 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8003b80 con 0x7f63f40feb80 2023-07-14T18:11:54.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.919+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f419ceb0 con 0x7f63f40feb80 2023-07-14T18:11:54.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.920+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f4106130 con 0x7f63f40feb80 2023-07-14T18:11:54.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.921+0000 7f63f1ffb700 1 -- 172.21.15.115:0/1193211307 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f63e8003ce0 con 0x7f63f40feb80 2023-07-14T18:11:54.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.922+0000 7f63f1ffb700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f63e40a11e0 0x7f63e40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:54.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.922+0000 7f63f1ffb700 1 -- 172.21.15.115:0/1193211307 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f63e80cc9e0 con 0x7f63f40feb80 2023-07-14T18:11:54.970 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.922+0000 7f63f3fff700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f63e40a11e0 0x7f63e40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:54.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.922+0000 7f63f3fff700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f63e40a11e0 0x7f63e40a3690 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f63e0005950 tx=0x7f63e000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:54.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:54.926+0000 7f63f1ffb700 1 -- 172.21.15.115:0/1193211307 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f63e8095cc0 con 0x7f63f40feb80 2023-07-14T18:11:55.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.118+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63f402cb50 con 0x7f63f40feb80 2023-07-14T18:11:55.159 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.119+0000 7f63f1ffb700 1 -- 172.21.15.115:0/1193211307 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63e80992e0 con 0x7f63f40feb80 2023-07-14T18:11:55.160 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:11:55.161 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f63e40a11e0 msgr2=0x7f63e40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f63e40a11e0 0x7f63e40a3690 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f63e0005950 tx=0x7f63e000a300 comp rx=0 tx=0).stop 2023-07-14T18:11:55.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f40feb80 msgr2=0x7f63f4194870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f40feb80 0x7f63f4194870 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c970 tx=0x7f63e800cc80 comp rx=0 tx=0).stop 2023-07-14T18:11:55.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 shutdown_connections 2023-07-14T18:11:55.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f63e40a11e0 0x7f63e40a3690 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f4100700 0x7f63f419be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63f40feb80 0x7f63f4194870 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 --2- 172.21.15.115:0/1193211307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63f40ffd70 0x7f63f4194db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 >> 172.21.15.115:0/1193211307 conn(0x7f63f40fa1a0 msgr2=0x7f63f4103270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:55.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 shutdown_connections 2023-07-14T18:11:55.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.121+0000 7f63fb0a4700 1 -- 172.21.15.115:0/1193211307 wait complete. 2023-07-14T18:11:55.174 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:11:55.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.257+0000 7ff3237e8700 1 -- 172.21.15.115:0/785201466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c0ffc10 msgr2=0x7ff31c100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.257+0000 7ff3237e8700 1 --2- 172.21.15.115:0/785201466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c0ffc10 0x7ff31c100020 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009990 tx=0x7ff30c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.257+0000 7ff3237e8700 1 -- 172.21.15.115:0/785201466 shutdown_connections 2023-07-14T18:11:55.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.257+0000 7ff3237e8700 1 --2- 172.21.15.115:0/785201466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c101790 0x7ff31c108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.257+0000 7ff3237e8700 1 --2- 172.21.15.115:0/785201466 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c0ffc10 0x7ff31c100020 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.257+0000 7ff3237e8700 1 --2- 172.21.15.115:0/785201466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff31c100e00 0x7ff31c101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.257+0000 7ff3237e8700 1 -- 172.21.15.115:0/785201466 >> 172.21.15.115:0/785201466 conn(0x7ff31c0fb250 msgr2=0x7ff31c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:55.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.257+0000 7ff3237e8700 1 -- 172.21.15.115:0/785201466 shutdown_connections 2023-07-14T18:11:55.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.257+0000 7ff3237e8700 1 -- 172.21.15.115:0/785201466 wait complete. 2023-07-14T18:11:55.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff3237e8700 1 Processor -- start 2023-07-14T18:11:55.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff3237e8700 1 -- start start 2023-07-14T18:11:55.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff3237e8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff31c0ffc10 0x7ff31c1958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff3237e8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c100e00 0x7ff31c195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff3237e8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c101790 0x7ff31c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff3237e8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c19d450 con 0x7ff31c101790 2023-07-14T18:11:55.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff3237e8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c19d590 con 0x7ff31c0ffc10 2023-07-14T18:11:55.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff3237e8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c19d720 con 0x7ff31c100e00 2023-07-14T18:11:55.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321d85700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c101790 0x7ff31c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321d85700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c101790 0x7ff31c19ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54088/0 (socket says 172.21.15.115:54088) 2023-07-14T18:11:55.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321d85700 1 -- 172.21.15.115:0/2467220793 learned_addr learned my addr 172.21.15.115:0/2467220793 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:55.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff320d83700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c100e00 0x7ff31c195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321584700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff31c0ffc10 0x7ff31c1958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321d85700 1 -- 172.21.15.115:0/2467220793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c100e00 msgr2=0x7ff31c195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321d85700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c100e00 0x7ff31c195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321d85700 1 -- 172.21.15.115:0/2467220793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff31c0ffc10 msgr2=0x7ff31c1958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321d85700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff31c0ffc10 0x7ff31c1958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321d85700 1 -- 172.21.15.115:0/2467220793 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c0095f0 con 0x7ff31c101790 2023-07-14T18:11:55.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321584700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff31c0ffc10 0x7ff31c1958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:55.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff321d85700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c101790 0x7ff31c19ce80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff314007ee0 tx=0x7ff31400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:55.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.258+0000 7ff3127fc700 1 -- 172.21.15.115:0/2467220793 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314017070 con 0x7ff31c101790 2023-07-14T18:11:55.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.259+0000 7ff3127fc700 1 -- 172.21.15.115:0/2467220793 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff31400f040 con 0x7ff31c101790 2023-07-14T18:11:55.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.259+0000 7ff3127fc700 1 -- 172.21.15.115:0/2467220793 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314005710 con 0x7ff31c101790 2023-07-14T18:11:55.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.259+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c19da00 con 0x7ff31c101790 2023-07-14T18:11:55.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.259+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c19dec0 con 0x7ff31c101790 2023-07-14T18:11:55.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.259+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31c10b2e0 con 0x7ff31c101790 2023-07-14T18:11:55.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.261+0000 7ff3127fc700 1 -- 172.21.15.115:0/2467220793 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff314005870 con 0x7ff31c101790 2023-07-14T18:11:55.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.261+0000 7ff3127fc700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff3080a1260 0x7ff3080a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.261+0000 7ff321584700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff3080a1260 0x7ff3080a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.261+0000 7ff3127fc700 1 -- 172.21.15.115:0/2467220793 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff3140ccc90 con 0x7ff31c101790 2023-07-14T18:11:55.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.262+0000 7ff321584700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff3080a1260 0x7ff3080a3710 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff30c006010 tx=0x7ff30c009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:55.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.264+0000 7ff3127fc700 1 -- 172.21.15.115:0/2467220793 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff314095f70 con 0x7ff31c101790 2023-07-14T18:11:55.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.454+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff31c106b80 con 0x7ff31c101790 2023-07-14T18:11:55.495 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.455+0000 7ff3127fc700 1 -- 172.21.15.115:0/2467220793 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff314099590 con 0x7ff31c101790 2023-07-14T18:11:55.496 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:11:55.497 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff3080a1260 msgr2=0x7ff3080a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff3080a1260 0x7ff3080a3710 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff30c006010 tx=0x7ff30c009530 comp rx=0 tx=0).stop 2023-07-14T18:11:55.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c101790 msgr2=0x7ff31c19ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c101790 0x7ff31c19ce80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff314007ee0 tx=0x7ff31400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 shutdown_connections 2023-07-14T18:11:55.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7ff3080a1260 0x7ff3080a3710 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff31c101790 0x7ff31c19ce80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.498 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff31c0ffc10 0x7ff31c1958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 --2- 172.21.15.115:0/2467220793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff31c100e00 0x7ff31c195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 >> 172.21.15.115:0/2467220793 conn(0x7ff31c0fb250 msgr2=0x7ff31c0fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:55.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 shutdown_connections 2023-07-14T18:11:55.499 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.457+0000 7ff3237e8700 1 -- 172.21.15.115:0/2467220793 wait complete. 2023-07-14T18:11:55.509 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:11:55.633 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 -- 172.21.15.115:0/2160655569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 msgr2=0x7fb744102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/2160655569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 0x7fb744102b10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb734009990 tx=0x7fb734009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 -- 172.21.15.115:0/2160655569 shutdown_connections 2023-07-14T18:11:55.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/2160655569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb744105970 0x7fb744107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/2160655569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 0x7fb744102b10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/2160655569 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb744103050 0x7fb744105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 -- 172.21.15.115:0/2160655569 >> 172.21.15.115:0/2160655569 conn(0x7fb7440fa130 msgr2=0x7fb7440fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:55.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 -- 172.21.15.115:0/2160655569 shutdown_connections 2023-07-14T18:11:55.634 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 -- 172.21.15.115:0/2160655569 wait complete. 2023-07-14T18:11:55.635 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 Processor -- start 2023-07-14T18:11:55.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.593+0000 7fb74b8bb700 1 -- start start 2023-07-14T18:11:55.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb74b8bb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 0x7fb744195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb74b8bb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb744103050 0x7fb744195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb74b8bb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb744105970 0x7fb74419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb74b8bb700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74419d3a0 con 0x7fb744100730 2023-07-14T18:11:55.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb74b8bb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74419d4e0 con 0x7fb744103050 2023-07-14T18:11:55.637 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb74b8bb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74419d670 con 0x7fb744105970 2023-07-14T18:11:55.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749657700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 0x7fb744195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749657700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 0x7fb744195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54098/0 (socket says 172.21.15.115:54098) 2023-07-14T18:11:55.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749657700 1 -- 172.21.15.115:0/360081906 learned_addr learned my addr 172.21.15.115:0/360081906 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:55.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749e58700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb744105970 0x7fb74419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb748e56700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb744103050 0x7fb744195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749657700 1 -- 172.21.15.115:0/360081906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb744105970 msgr2=0x7fb74419cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749657700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb744105970 0x7fb74419cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749657700 1 -- 172.21.15.115:0/360081906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb744103050 msgr2=0x7fb744195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749657700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb744103050 0x7fb744195d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749657700 1 -- 172.21.15.115:0/360081906 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7340095f0 con 0x7fb744100730 2023-07-14T18:11:55.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749e58700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb744105970 0x7fb74419cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:55.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb749657700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 0x7fb744195820 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb734006010 tx=0x7fb734004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:55.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb73a7fc700 1 -- 172.21.15.115:0/360081906 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73401d070 con 0x7fb744100730 2023-07-14T18:11:55.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb73a7fc700 1 -- 172.21.15.115:0/360081906 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb7340041b0 con 0x7fb744100730 2023-07-14T18:11:55.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74419d8f0 con 0x7fb744100730 2023-07-14T18:11:55.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb73a7fc700 1 -- 172.21.15.115:0/360081906 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734022da0 con 0x7fb744100730 2023-07-14T18:11:55.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.594+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74419dd10 con 0x7fb744100730 2023-07-14T18:11:55.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.595+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7440fbca0 con 0x7fb744100730 2023-07-14T18:11:55.640 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.596+0000 7fb73a7fc700 1 -- 172.21.15.115:0/360081906 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fb734022570 con 0x7fb744100730 2023-07-14T18:11:55.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.597+0000 7fb73a7fc700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb7300a11e0 0x7fb7300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.597+0000 7fb748e56700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb7300a11e0 0x7fb7300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.597+0000 7fb73a7fc700 1 -- 172.21.15.115:0/360081906 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb7340ce6f0 con 0x7fb744100730 2023-07-14T18:11:55.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.597+0000 7fb748e56700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb7300a11e0 0x7fb7300a3690 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb740005950 tx=0x7fb7400058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:55.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.600+0000 7fb73a7fc700 1 -- 172.21.15.115:0/360081906 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb734097a60 con 0x7fb744100730 2023-07-14T18:11:55.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.786+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb74402ca70 con 0x7fb744100730 2023-07-14T18:11:55.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.786+0000 7fb73a7fc700 1 -- 172.21.15.115:0/360081906 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb73409b080 con 0x7fb744100730 2023-07-14T18:11:55.827 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 64 2023-07-14T18:11:55.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.788+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb7300a11e0 msgr2=0x7fb7300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.788+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb7300a11e0 0x7fb7300a3690 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb740005950 tx=0x7fb7400058e0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.788+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 msgr2=0x7fb744195820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.788+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 0x7fb744195820 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb734006010 tx=0x7fb734004730 comp rx=0 tx=0).stop 2023-07-14T18:11:55.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.788+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 shutdown_connections 2023-07-14T18:11:55.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.788+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fb7300a11e0 0x7fb7300a3690 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.789+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb744105970 0x7fb74419cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.789+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb744100730 0x7fb744195820 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.789+0000 7fb74b8bb700 1 --2- 172.21.15.115:0/360081906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb744103050 0x7fb744195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.789+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 >> 172.21.15.115:0/360081906 conn(0x7fb7440fa130 msgr2=0x7fb744101390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:55.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.789+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 shutdown_connections 2023-07-14T18:11:55.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.789+0000 7fb74b8bb700 1 -- 172.21.15.115:0/360081906 wait complete. 2023-07-14T18:11:55.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:55 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2156424937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:55.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:55 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3973690026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:55.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:55 smithi115 ceph-mon[127451]: pgmap v302: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T18:11:55.840 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:55 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1193211307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:55.841 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:11:55.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:55 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2156424937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:55.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:55 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3973690026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:55.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:55 smithi150 ceph-mon[128853]: pgmap v302: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T18:11:55.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:55 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1193211307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:55.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:55 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2156424937' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:55.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:55 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3973690026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:55.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:55 smithi187 ceph-mon[129317]: pgmap v302: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T18:11:55.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:55 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1193211307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:11:55.967 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.927+0000 7f5185abf700 1 -- 172.21.15.115:0/1105126413 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5180101790 msgr2=0x7f5180108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.927+0000 7f5185abf700 1 --2- 172.21.15.115:0/1105126413 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5180101790 0x7f5180108050 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5170009990 tx=0x7f5170009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.927+0000 7f5185abf700 1 -- 172.21.15.115:0/1105126413 shutdown_connections 2023-07-14T18:11:55.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.927+0000 7f5185abf700 1 --2- 172.21.15.115:0/1105126413 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5180101790 0x7f5180108050 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.927+0000 7f5185abf700 1 --2- 172.21.15.115:0/1105126413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51800ffc10 0x7f5180100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.927+0000 7f5185abf700 1 --2- 172.21.15.115:0/1105126413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5180100e00 0x7f5180101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.968 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.927+0000 7f5185abf700 1 -- 172.21.15.115:0/1105126413 >> 172.21.15.115:0/1105126413 conn(0x7f51800fb250 msgr2=0x7f51800fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:55.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.927+0000 7f5185abf700 1 -- 172.21.15.115:0/1105126413 shutdown_connections 2023-07-14T18:11:55.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.927+0000 7f5185abf700 1 -- 172.21.15.115:0/1105126413 wait complete. 2023-07-14T18:11:55.969 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f5185abf700 1 Processor -- start 2023-07-14T18:11:55.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f5185abf700 1 -- start start 2023-07-14T18:11:55.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f5185abf700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51800ffc10 0x7f51800743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f5185abf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5180100e00 0x7f5180072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f5185abf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5180101790 0x7f5180072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.971 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f5185abf700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5180073710 con 0x7f51800ffc10 2023-07-14T18:11:55.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f5185abf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5180073880 con 0x7f5180100e00 2023-07-14T18:11:55.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f5185abf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518019f8a0 con 0x7f5180101790 2023-07-14T18:11:55.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51800ffc10 0x7f51800743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51800ffc10 0x7f51800743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54112/0 (socket says 172.21.15.115:54112) 2023-07-14T18:11:55.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517f7fe700 1 -- 172.21.15.115:0/1776987795 learned_addr learned my addr 172.21.15.115:0/1776987795 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:55.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517effd700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5180100e00 0x7f5180072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517ffff700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5180101790 0x7f5180072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.972 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517f7fe700 1 -- 172.21.15.115:0/1776987795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5180101790 msgr2=0x7f5180072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517f7fe700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5180101790 0x7f5180072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517f7fe700 1 -- 172.21.15.115:0/1776987795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5180100e00 msgr2=0x7f5180072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:55.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517f7fe700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5180100e00 0x7f5180072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:55.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517f7fe700 1 -- 172.21.15.115:0/1776987795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51700095f0 con 0x7f51800ffc10 2023-07-14T18:11:55.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517ffff700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5180101790 0x7f5180072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:55.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.928+0000 7f517f7fe700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51800ffc10 0x7f51800743f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5178007f10 tx=0x7f517800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:55.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.929+0000 7f517cff9700 1 -- 172.21.15.115:0/1776987795 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5178017070 con 0x7f51800ffc10 2023-07-14T18:11:55.973 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.929+0000 7f517cff9700 1 -- 172.21.15.115:0/1776987795 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f517800f040 con 0x7f51800ffc10 2023-07-14T18:11:55.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.929+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518019fb80 con 0x7f51800ffc10 2023-07-14T18:11:55.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.929+0000 7f517cff9700 1 -- 172.21.15.115:0/1776987795 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5178004500 con 0x7f51800ffc10 2023-07-14T18:11:55.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.929+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51801a0090 con 0x7f51800ffc10 2023-07-14T18:11:55.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.930+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51801071f0 con 0x7f51800ffc10 2023-07-14T18:11:55.974 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.931+0000 7f517cff9700 1 -- 172.21.15.115:0/1776987795 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f5178004660 con 0x7f51800ffc10 2023-07-14T18:11:55.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.931+0000 7f517cff9700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51680a11e0 0x7f51680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:55.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.931+0000 7f517cff9700 1 -- 172.21.15.115:0/1776987795 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f51780ccb80 con 0x7f51800ffc10 2023-07-14T18:11:55.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.931+0000 7f517effd700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51680a11e0 0x7f51680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:55.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.932+0000 7f517effd700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51680a11e0 0x7f51680a3690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5174005950 tx=0x7f517400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:55.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:55.935+0000 7f517cff9700 1 -- 172.21.15.115:0/1776987795 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5178095e60 con 0x7f51800ffc10 2023-07-14T18:11:56.162 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.122+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f518002cb30 con 0x7f51800ffc10 2023-07-14T18:11:56.163 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.122+0000 7f517cff9700 1 -- 172.21.15.115:0/1776987795 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5178099480 con 0x7f51800ffc10 2023-07-14T18:11:56.163 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:11:56.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51680a11e0 msgr2=0x7f51680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51680a11e0 0x7f51680a3690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5174005950 tx=0x7f517400a300 comp rx=0 tx=0).stop 2023-07-14T18:11:56.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51800ffc10 msgr2=0x7f51800743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51800ffc10 0x7f51800743f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5178007f10 tx=0x7f517800d3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.165 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 shutdown_connections 2023-07-14T18:11:56.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f51680a11e0 0x7f51680a3690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5180101790 0x7f5180072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f51800ffc10 0x7f51800743f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 --2- 172.21.15.115:0/1776987795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5180100e00 0x7f5180072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 >> 172.21.15.115:0/1776987795 conn(0x7f51800fb250 msgr2=0x7f5180104330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:56.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 shutdown_connections 2023-07-14T18:11:56.166 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.124+0000 7f5185abf700 1 -- 172.21.15.115:0/1776987795 wait complete. 2023-07-14T18:11:56.177 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:11:56.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.257+0000 7f863f1c9700 1 -- 172.21.15.115:0/76451677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8638100690 msgr2=0x7f8638106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.257+0000 7f863f1c9700 1 --2- 172.21.15.115:0/76451677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8638100690 0x7f8638106f50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f863000b0a0 tx=0x7f863000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.257+0000 7f863f1c9700 1 -- 172.21.15.115:0/76451677 shutdown_connections 2023-07-14T18:11:56.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.257+0000 7f863f1c9700 1 --2- 172.21.15.115:0/76451677 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8638100690 0x7f8638106f50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.257+0000 7f863f1c9700 1 --2- 172.21.15.115:0/76451677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86380feb10 0x7f86380fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.257+0000 7f863f1c9700 1 --2- 172.21.15.115:0/76451677 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86380ffd00 0x7f8638100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.257+0000 7f863f1c9700 1 -- 172.21.15.115:0/76451677 >> 172.21.15.115:0/76451677 conn(0x7f86380fa130 msgr2=0x7f86380fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:56.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.257+0000 7f863f1c9700 1 -- 172.21.15.115:0/76451677 shutdown_connections 2023-07-14T18:11:56.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.257+0000 7f863f1c9700 1 -- 172.21.15.115:0/76451677 wait complete. 2023-07-14T18:11:56.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863f1c9700 1 Processor -- start 2023-07-14T18:11:56.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863f1c9700 1 -- start start 2023-07-14T18:11:56.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863f1c9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86380feb10 0x7f8638069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:56.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863f1c9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86380ffd00 0x7f863819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:56.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863f1c9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8638100690 0x7f863819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:56.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863f1c9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863819d3d0 con 0x7f86380feb10 2023-07-14T18:11:56.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863f1c9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863819d540 con 0x7f8638100690 2023-07-14T18:11:56.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863f1c9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863819d700 con 0x7f86380ffd00 2023-07-14T18:11:56.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863cf65700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86380feb10 0x7f8638069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:56.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863cf65700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86380feb10 0x7f8638069550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54126/0 (socket says 172.21.15.115:54126) 2023-07-14T18:11:56.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863cf65700 1 -- 172.21.15.115:0/3169433311 learned_addr learned my addr 172.21.15.115:0/3169433311 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:56.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.258+0000 7f863d766700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8638100690 0x7f863819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:56.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f8637fff700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86380ffd00 0x7f863819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:56.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f863cf65700 1 -- 172.21.15.115:0/3169433311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86380ffd00 msgr2=0x7f863819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f863cf65700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86380ffd00 0x7f863819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.302 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f863cf65700 1 -- 172.21.15.115:0/3169433311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8638100690 msgr2=0x7f863819cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f863cf65700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8638100690 0x7f863819cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f863cf65700 1 -- 172.21.15.115:0/3169433311 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630009d00 con 0x7f86380feb10 2023-07-14T18:11:56.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f8637fff700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86380ffd00 0x7f863819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:56.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f863cf65700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86380feb10 0x7f8638069550 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f862c00d980 tx=0x7f862c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:56.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f8635ffb700 1 -- 172.21.15.115:0/3169433311 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c004020 con 0x7f86380feb10 2023-07-14T18:11:56.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f8635ffb700 1 -- 172.21.15.115:0/3169433311 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f862c009730 con 0x7f86380feb10 2023-07-14T18:11:56.303 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863819d9e0 con 0x7f86380feb10 2023-07-14T18:11:56.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f8635ffb700 1 -- 172.21.15.115:0/3169433311 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c003a00 con 0x7f86380feb10 2023-07-14T18:11:56.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.259+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863819def0 con 0x7f86380feb10 2023-07-14T18:11:56.304 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.260+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863810a1a0 con 0x7f86380feb10 2023-07-14T18:11:56.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.261+0000 7f8635ffb700 1 -- 172.21.15.115:0/3169433311 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f862c0098a0 con 0x7f86380feb10 2023-07-14T18:11:56.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.262+0000 7f8635ffb700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f86280a1220 0x7f86280a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:56.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.262+0000 7f8635ffb700 1 -- 172.21.15.115:0/3169433311 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f862c0ccae0 con 0x7f86380feb10 2023-07-14T18:11:56.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.262+0000 7f8637fff700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f86280a1220 0x7f86280a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:56.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.262+0000 7f8637fff700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f86280a1220 0x7f86280a36d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8624005fd0 tx=0x7f8624005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:56.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.266+0000 7f8635ffb700 1 -- 172.21.15.115:0/3169433311 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f862c095dc0 con 0x7f86380feb10 2023-07-14T18:11:56.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.497+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f863810a3c0 con 0x7f86380feb10 2023-07-14T18:11:56.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.497+0000 7f8635ffb700 1 -- 172.21.15.115:0/3169433311 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10851 (secure 0 0 0) 0x7f862c0993e0 con 0x7f86380feb10 2023-07-14T18:11:56.538 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:56.538 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:10:38.422616+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24493,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24446":{"gid":24446,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:11:56.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.499+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f86280a1220 msgr2=0x7f86280a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.499+0000 7f863f1c9700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f86280a1220 0x7f86280a36d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8624005fd0 tx=0x7f8624005e20 comp rx=0 tx=0).stop 2023-07-14T18:11:56.540 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.499+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86380feb10 msgr2=0x7f8638069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.499+0000 7f863f1c9700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86380feb10 0x7f8638069550 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f862c00d980 tx=0x7f862c00dc90 comp rx=0 tx=0).stop 2023-07-14T18:11:56.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.500+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 shutdown_connections 2023-07-14T18:11:56.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.500+0000 7f863f1c9700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f86280a1220 0x7f86280a36d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.500+0000 7f863f1c9700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8638100690 0x7f863819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.500+0000 7f863f1c9700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f86380feb10 0x7f8638069550 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.500+0000 7f863f1c9700 1 --2- 172.21.15.115:0/3169433311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86380ffd00 0x7f863819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.541 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.500+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 >> 172.21.15.115:0/3169433311 conn(0x7f86380fa130 msgr2=0x7f86380fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:56.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.500+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 shutdown_connections 2023-07-14T18:11:56.542 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.500+0000 7f863f1c9700 1 -- 172.21.15.115:0/3169433311 wait complete. 2023-07-14T18:11:56.542 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 24 2023-07-14T18:11:56.552 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:11:56.673 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.633+0000 7fe31ba4b700 1 -- 172.21.15.115:0/4019424695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3141058c0 msgr2=0x7fe314107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.633+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/4019424695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3141058c0 0x7fe314107d20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00b0a0 tx=0x7fe30c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.633+0000 7fe31ba4b700 1 -- 172.21.15.115:0/4019424695 shutdown_connections 2023-07-14T18:11:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.633+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/4019424695 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe3141058c0 0x7fe314107d20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.633+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/4019424695 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe314100680 0x7fe314102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.633+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/4019424695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe314102fa0 0x7fe314105380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.633+0000 7fe31ba4b700 1 -- 172.21.15.115:0/4019424695 >> 172.21.15.115:0/4019424695 conn(0x7fe3140fa100 msgr2=0x7fe3140fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:56.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.633+0000 7fe31ba4b700 1 -- 172.21.15.115:0/4019424695 shutdown_connections 2023-07-14T18:11:56.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.633+0000 7fe31ba4b700 1 -- 172.21.15.115:0/4019424695 wait complete. 2023-07-14T18:11:56.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe31ba4b700 1 Processor -- start 2023-07-14T18:11:56.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe31ba4b700 1 -- start start 2023-07-14T18:11:56.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe31ba4b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe314100680 0x7fe3141958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:56.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe31ba4b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe314102fa0 0x7fe314195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:56.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe31ba4b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3141058c0 0x7fe31419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:56.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe31ba4b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419d460 con 0x7fe314100680 2023-07-14T18:11:56.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe31ba4b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419d5a0 con 0x7fe314102fa0 2023-07-14T18:11:56.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe31ba4b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419d730 con 0x7fe3141058c0 2023-07-14T18:11:56.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe3197e7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe314100680 0x7fe3141958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:56.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe3197e7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe314100680 0x7fe3141958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54140/0 (socket says 172.21.15.115:54140) 2023-07-14T18:11:56.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe3197e7700 1 -- 172.21.15.115:0/3467032915 learned_addr learned my addr 172.21.15.115:0/3467032915 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:56.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe319fe8700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3141058c0 0x7fe31419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:56.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe318fe6700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe314102fa0 0x7fe314195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:56.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe3197e7700 1 -- 172.21.15.115:0/3467032915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3141058c0 msgr2=0x7fe31419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe3197e7700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3141058c0 0x7fe31419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe3197e7700 1 -- 172.21.15.115:0/3467032915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe314102fa0 msgr2=0x7fe314195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe3197e7700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe314102fa0 0x7fe314195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe3197e7700 1 -- 172.21.15.115:0/3467032915 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c009d00 con 0x7fe314100680 2023-07-14T18:11:56.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe318fe6700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe314102fa0 0x7fe314195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:56.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.634+0000 7fe319fe8700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3141058c0 0x7fe31419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:56.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.635+0000 7fe3197e7700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe314100680 0x7fe3141958e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe31000d950 tx=0x7fe31000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:56.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.635+0000 7fe30a7fc700 1 -- 172.21.15.115:0/3467032915 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310009b40 con 0x7fe314100680 2023-07-14T18:11:56.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.635+0000 7fe30a7fc700 1 -- 172.21.15.115:0/3467032915 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe310009ca0 con 0x7fe314100680 2023-07-14T18:11:56.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.635+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31419da10 con 0x7fe314100680 2023-07-14T18:11:56.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.635+0000 7fe30a7fc700 1 -- 172.21.15.115:0/3467032915 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31000f620 con 0x7fe314100680 2023-07-14T18:11:56.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.635+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31419df70 con 0x7fe314100680 2023-07-14T18:11:56.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.636+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3140fbc70 con 0x7fe314100680 2023-07-14T18:11:56.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.637+0000 7fe30a7fc700 1 -- 172.21.15.115:0/3467032915 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fe310010460 con 0x7fe314100680 2023-07-14T18:11:56.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.637+0000 7fe30a7fc700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe3000a12a0 0x7fe3000a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:56.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.637+0000 7fe318fe6700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe3000a12a0 0x7fe3000a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:56.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.637+0000 7fe30a7fc700 1 -- 172.21.15.115:0/3467032915 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe3100cd0f0 con 0x7fe314100680 2023-07-14T18:11:56.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.638+0000 7fe318fe6700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe3000a12a0 0x7fe3000a3750 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe304005950 tx=0x7fe304005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:56.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.641+0000 7fe30a7fc700 1 -- 172.21.15.115:0/3467032915 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe3100963d0 con 0x7fe314100680 2023-07-14T18:11:56.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:56 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2467220793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:56.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:56 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/360081906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:56.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:56 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1776987795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:56.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.832+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe31402c8e0 con 0x7fe314100680 2023-07-14T18:11:56.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:56 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2467220793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:56.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:56 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/360081906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:56.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:56 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1776987795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:56.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.833+0000 7fe30a7fc700 1 -- 172.21.15.115:0/3467032915 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24558 (secure 0 0 0) 0x7fe310016090 con 0x7fe314100680 2023-07-14T18:11:56.874 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:11:56.875 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":89,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:11:30.172466+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:11:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe3000a12a0 msgr2=0x7fe3000a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe3000a12a0 0x7fe3000a3750 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe304005950 tx=0x7fe304005f50 comp rx=0 tx=0).stop 2023-07-14T18:11:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe314100680 msgr2=0x7fe3141958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe314100680 0x7fe3141958e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe31000d950 tx=0x7fe31000dd10 comp rx=0 tx=0).stop 2023-07-14T18:11:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 shutdown_connections 2023-07-14T18:11:56.876 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe3000a12a0 0x7fe3000a3750 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe3141058c0 0x7fe31419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe314100680 0x7fe3141958e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 --2- 172.21.15.115:0/3467032915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe314102fa0 0x7fe314195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:56.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.835+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 >> 172.21.15.115:0/3467032915 conn(0x7fe3140fa100 msgr2=0x7fe3141012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:56.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.836+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 shutdown_connections 2023-07-14T18:11:56.877 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.836+0000 7fe31ba4b700 1 -- 172.21.15.115:0/3467032915 wait complete. 2023-07-14T18:11:56.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T18:10:17.634244+0000', 'modified': '2023-07-14T18:10:38.422616+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24515, 'mds_1': 24493, 'mds_2': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24446': {'gid': 24446, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6835/3630781789', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 3630781789}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 3630781789}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.187:6835/4258537735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6834', 'nonce': 4258537735}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 4258537735}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24475': {'gid': 24475, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.187:6837/771796341', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6836', 'nonce': 771796341}, {'type': 'v1', 'addr': '172.21.15.187:6837', 'nonce': 771796341}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6837/1462100932', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6836', 'nonce': 1462100932}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 1462100932}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.187:6839/1449243194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6838', 'nonce': 1449243194}, {'type': 'v1', 'addr': '172.21.15.187:6839', 'nonce': 1449243194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6839/2895627668', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 2895627668}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 2895627668}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T18:11:56.889 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-14T18:11:56.889 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-07-14T18:11:56.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:56 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2467220793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:11:56.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:56 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/360081906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:11:56.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:56 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1776987795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:11:57.013 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.971+0000 7f6e85207700 1 -- 172.21.15.115:0/1953339841 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e801006f0 msgr2=0x7f6e80106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.971+0000 7f6e85207700 1 --2- 172.21.15.115:0/1953339841 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e801006f0 0x7f6e80106fb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800b0a0 tx=0x7f6e7800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.971+0000 7f6e85207700 1 -- 172.21.15.115:0/1953339841 shutdown_connections 2023-07-14T18:11:57.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.971+0000 7f6e85207700 1 --2- 172.21.15.115:0/1953339841 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e801006f0 0x7f6e80106fb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.015 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.971+0000 7f6e85207700 1 --2- 172.21.15.115:0/1953339841 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e800feb70 0x7f6e800fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.971+0000 7f6e85207700 1 --2- 172.21.15.115:0/1953339841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e800ffd60 0x7f6e801001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.971+0000 7f6e85207700 1 -- 172.21.15.115:0/1953339841 >> 172.21.15.115:0/1953339841 conn(0x7f6e800fa1d0 msgr2=0x7f6e800fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.971+0000 7f6e85207700 1 -- 172.21.15.115:0/1953339841 shutdown_connections 2023-07-14T18:11:57.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.971+0000 7f6e85207700 1 -- 172.21.15.115:0/1953339841 wait complete. 2023-07-14T18:11:57.016 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e85207700 1 Processor -- start 2023-07-14T18:11:57.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e85207700 1 -- start start 2023-07-14T18:11:57.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e85207700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e800feb70 0x7f6e80069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e85207700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e800ffd60 0x7f6e8019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e85207700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e801006f0 0x7f6e8019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e85207700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8019d490 con 0x7f6e800ffd60 2023-07-14T18:11:57.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e85207700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8019d600 con 0x7f6e800feb70 2023-07-14T18:11:57.017 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e85207700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8019d7c0 con 0x7f6e801006f0 2023-07-14T18:11:57.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e7e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e800ffd60 0x7f6e8019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e7e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e800ffd60 0x7f6e8019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54154/0 (socket says 172.21.15.115:54154) 2023-07-14T18:11:57.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e7e59c700 1 -- 172.21.15.115:0/3395429664 learned_addr learned my addr 172.21.15.115:0/3395429664 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e7f59e700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e801006f0 0x7f6e8019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.972+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e800feb70 0x7f6e80069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e7e59c700 1 -- 172.21.15.115:0/3395429664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e801006f0 msgr2=0x7f6e8019ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.018 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e801006f0 0x7f6e8019ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e7e59c700 1 -- 172.21.15.115:0/3395429664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e800feb70 msgr2=0x7f6e80069560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e800feb70 0x7f6e80069560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e7e59c700 1 -- 172.21.15.115:0/3395429664 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e78009d00 con 0x7f6e800ffd60 2023-07-14T18:11:57.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e7f59e700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e801006f0 0x7f6e8019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:57.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e800feb70 0x7f6e80069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:57.019 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e800ffd60 0x7f6e8019a6b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000c9b0 tx=0x7f6e7000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e77fff700 1 -- 172.21.15.115:0/3395429664 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70007b00 con 0x7f6e800ffd60 2023-07-14T18:11:57.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e85207700 1 -- 172.21.15.115:0/3395429664 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8019daa0 con 0x7f6e800ffd60 2023-07-14T18:11:57.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e77fff700 1 -- 172.21.15.115:0/3395429664 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6e70007c60 con 0x7f6e800ffd60 2023-07-14T18:11:57.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e77fff700 1 -- 172.21.15.115:0/3395429664 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70018660 con 0x7f6e800ffd60 2023-07-14T18:11:57.020 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.973+0000 7f6e85207700 1 -- 172.21.15.115:0/3395429664 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8019dfb0 con 0x7f6e800ffd60 2023-07-14T18:11:57.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.975+0000 7f6e77fff700 1 -- 172.21.15.115:0/3395429664 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f6e7000f450 con 0x7f6e800ffd60 2023-07-14T18:11:57.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.976+0000 7f6e77fff700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6e6c0a11e0 0x7f6e6c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.976+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6e6c0a11e0 0x7f6e6c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.976+0000 7f6e77fff700 1 -- 172.21.15.115:0/3395429664 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6e700cce70 con 0x7f6e800ffd60 2023-07-14T18:11:57.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.976+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6e6c0a11e0 0x7f6e6c0a3690 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6e800ffbc0 tx=0x7f6e68008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.025 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e9139fb80 1 Processor -- start 2023-07-14T18:11:57.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e9139fb80 1 -- start start 2023-07-14T18:11:57.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 0x55bdb8aab320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8aabe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aac460 0x55bdb8aac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb89afc60 con 0x55bdb8aab860 2023-07-14T18:11:57.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb8a029f0 con 0x55bdb8aac460 2023-07-14T18:11:57.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb89fbe80 con 0x55bdb8aaaf10 2023-07-14T18:11:57.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8aabe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8aabe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54162/0 (socket says 172.21.15.115:54162) 2023-07-14T18:11:57.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7e59c700 1 -- 172.21.15.115:0/813857176 learned_addr learned my addr 172.21.15.115:0/813857176 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7f59e700 1 --2- 172.21.15.115:0/813857176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aac460 0x55bdb8aac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7e59c700 1 -- 172.21.15.115:0/813857176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 msgr2=0x55bdb8aab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/813857176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 0x55bdb8aab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/813857176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 0x55bdb8aab320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.027 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7e59c700 1 -- 172.21.15.115:0/813857176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aac460 msgr2=0x55bdb8aac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/813857176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aac460 0x55bdb8aac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.977+0000 7f6e7e59c700 1 -- 172.21.15.115:0/813857176 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdb8829d90 con 0x55bdb8aab860 2023-07-14T18:11:57.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.978+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/813857176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8aabe80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6e70096310 tx=0x7f6e70099a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f1defd0906423e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.978+0000 7f6e757fa700 1 -- 172.21.15.115:0/813857176 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7009de40 con 0x55bdb8aab860 2023-07-14T18:11:57.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.978+0000 7f6e757fa700 1 -- 172.21.15.115:0/813857176 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6e700a2260 con 0x55bdb8aab860 2023-07-14T18:11:57.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.978+0000 7f6e757fa700 1 -- 172.21.15.115:0/813857176 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7009a070 con 0x55bdb8aab860 2023-07-14T18:11:57.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.978+0000 7f6e9139fb80 1 -- 172.21.15.115:0/813857176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 msgr2=0x55bdb8aabe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.978+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/813857176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8aabe80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6e70096310 tx=0x7f6e70099a60 comp rx=0 tx=0).stop 2023-07-14T18:11:57.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.978+0000 7f6e9139fb80 1 -- 172.21.15.115:0/813857176 shutdown_connections 2023-07-14T18:11:57.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/813857176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aac460 0x55bdb8aac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/813857176 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 0x55bdb8aab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/813857176 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8aabe80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 -- 172.21.15.115:0/813857176 >> 172.21.15.115:0/813857176 conn(0x55bdb8aa0630 msgr2=0x55bdb8aa5980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 -- 172.21.15.115:0/813857176 shutdown_connections 2023-07-14T18:11:57.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 -- 172.21.15.115:0/813857176 wait complete. 2023-07-14T18:11:57.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 -- start start 2023-07-14T18:11:57.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 Processor -- start 2023-07-14T18:11:57.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ab49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aab860 0x55bdb8ab4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aac460 0x55bdb8abbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb899ba70 con 0x55bdb8aac460 2023-07-14T18:11:57.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb899b770 con 0x55bdb8aab860 2023-07-14T18:11:57.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb887f8d0 con 0x55bdb8aaaf10 2023-07-14T18:11:57.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e9139fb80 10 client.0 _ll_register_callbacks cb 0x55bdb8999710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:57.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e7f59e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aac460 0x55bdb8abbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e7f59e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aac460 0x55bdb8abbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54170/0 (socket says 172.21.15.115:54170) 2023-07-14T18:11:57.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e7f59e700 1 -- 172.21.15.115:0/2536197996 learned_addr learned my addr 172.21.15.115:0/2536197996 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.979+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aab860 0x55bdb8ab4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e7f59e700 1 -- 172.21.15.115:0/2536197996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 msgr2=0x55bdb8ab49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.031 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ab49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e7f59e700 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ab49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e7f59e700 1 -- 172.21.15.115:0/2536197996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aab860 msgr2=0x55bdb8ab4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e7f59e700 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aab860 0x55bdb8ab4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e7f59e700 1 -- 172.21.15.115:0/2536197996 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8019dfb0 con 0x55bdb8aac460 2023-07-14T18:11:57.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aab860 0x55bdb8ab4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:57.032 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ab49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:57.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e7f59e700 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aac460 0x55bdb8abbf20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009440 tx=0x7f6e78009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ba82290e8543d3 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e667fc700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:57.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e667fc700 1 -- 172.21.15.115:0/2536197996 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7800e040 con 0x55bdb8aac460 2023-07-14T18:11:57.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e667fc700 1 -- 172.21.15.115:0/2536197996 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6e7801c640 con 0x55bdb8aac460 2023-07-14T18:11:57.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e9139fb80 1 -- 172.21.15.115:0/2536197996 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bdb89554a0 con 0x55bdb8aac460 2023-07-14T18:11:57.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e667fc700 1 -- 172.21.15.115:0/2536197996 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e78004ea0 con 0x55bdb8aac460 2023-07-14T18:11:57.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e9139fb80 10 client.15393 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:57.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e9139fb80 1 -- 172.21.15.115:0/2536197996 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bdb8812c10 con 0x55bdb8aac460 2023-07-14T18:11:57.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e667fc700 1 -- 172.21.15.115:0/2536197996 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55bdb89554a0 con 0x55bdb8aac460 2023-07-14T18:11:57.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e667fc700 1 -- 172.21.15.115:0/2536197996 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6e78019050 con 0x55bdb8aac460 2023-07-14T18:11:57.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e9139fb80 10 client.15393 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:57.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e9139fb80 10 client.15393 resolve_mds: resolved name 'f' to daemon mds.f 2023-07-14T18:11:57.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e9139fb80 20 client.15393 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:57.034 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55bdb8abfbf0 0x55bdb8abefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e9139fb80 4 client.15393 mds_command: new command op to 24463 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:11:57.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.980+0000 7f6e9139fb80 1 -- 172.21.15.115:0/2536197996 --> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bdb8a79d50 con 0x55bdb8abfbf0 2023-07-14T18:11:57.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.981+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55bdb8abfbf0 0x55bdb8abefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.981+0000 7f6e667fc700 10 client.15393 ms_handle_connect on v2:172.21.15.187:6834/4258537735 2023-07-14T18:11:57.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.981+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55bdb8abfbf0 0x55bdb8abefd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f86924a99c2b2670 server_cookie=e8a75439e07bc78c in_seq=0 out_seq=0 2023-07-14T18:11:57.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e667fc700 1 -- 172.21.15.115:0/2536197996 <== mds.2 v2:172.21.15.187:6834/4258537735 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55bdb8a79d50 con 0x55bdb8abfbf0 2023-07-14T18:11:57.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e667fc700 10 client.15393 handle_command_reply: tid=0 2023-07-14T18:11:57.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 client.15393 shutdown 2023-07-14T18:11:57.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 -- 172.21.15.115:0/2536197996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aac460 msgr2=0x55bdb8abbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aac460 0x55bdb8abbf20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009440 tx=0x7f6e78009520 comp rx=0 tx=0).stop 2023-07-14T18:11:57.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 -- 172.21.15.115:0/2536197996 shutdown_connections 2023-07-14T18:11:57.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55bdb8abfbf0 0x55bdb8abefd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aac460 0x55bdb8abbf20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e667fc700 0 client.15393 ms_handle_reset on v2:172.21.15.187:6834/4258537735 2023-07-14T18:11:57.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ab49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/2536197996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aab860 0x55bdb8ab4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 -- 172.21.15.115:0/2536197996 >> 172.21.15.115:0/2536197996 conn(0x55bdb8aa0630 msgr2=0x55bdb8aa4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 -- 172.21.15.115:0/2536197996 shutdown_connections 2023-07-14T18:11:57.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 1 -- 172.21.15.115:0/2536197996 wait complete. 2023-07-14T18:11:57.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.982+0000 7f6e9139fb80 20 client.15393 trim_cache size 0 max 16384 2023-07-14T18:11:57.039 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.999+0000 7f6e9139fb80 1 Processor -- start 2023-07-14T18:11:57.040 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:56.999+0000 7f6e9139fb80 1 -- start start 2023-07-14T18:11:57.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ab2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aac460 0x55bdb8ab3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb8ab4570 con 0x55bdb8aab860 2023-07-14T18:11:57.041 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb8ab5b90 con 0x55bdb8aaaf10 2023-07-14T18:11:57.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb89ca790 con 0x55bdb8aac460 2023-07-14T18:11:57.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e7e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e7e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54176/0 (socket says 172.21.15.115:54176) 2023-07-14T18:11:57.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e7e59c700 1 -- 172.21.15.115:0/3449543895 learned_addr learned my addr 172.21.15.115:0/3449543895 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e7e59c700 1 -- 172.21.15.115:0/3449543895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aac460 msgr2=0x55bdb8ab3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/3449543895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aac460 0x55bdb8ab3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e7e59c700 1 -- 172.21.15.115:0/3449543895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 msgr2=0x55bdb8ab2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.042 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/3449543895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ab2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e7e59c700 1 -- 172.21.15.115:0/3449543895 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdb8aa5f00 con 0x55bdb8aab860 2023-07-14T18:11:57.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/3449543895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac8960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000afd0 tx=0x7f6e700ca760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45290f3e80acb11b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e75ffb700 1 -- 172.21.15.115:0/3449543895 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7009de40 con 0x55bdb8aab860 2023-07-14T18:11:57.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e75ffb700 1 -- 172.21.15.115:0/3449543895 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6e700981b0 con 0x55bdb8aab860 2023-07-14T18:11:57.043 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.000+0000 7f6e75ffb700 1 -- 172.21.15.115:0/3449543895 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7000f940 con 0x55bdb8aab860 2023-07-14T18:11:57.046 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 -- 172.21.15.115:0/3449543895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 msgr2=0x55bdb8ac8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/3449543895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac8960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000afd0 tx=0x7f6e700ca760 comp rx=0 tx=0).stop 2023-07-14T18:11:57.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 -- 172.21.15.115:0/3449543895 shutdown_connections 2023-07-14T18:11:57.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/3449543895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aac460 0x55bdb8ab3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/3449543895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ab2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/3449543895 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac8960 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.047 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 -- 172.21.15.115:0/3449543895 >> 172.21.15.115:0/3449543895 conn(0x55bdb8aa0630 msgr2=0x55bdb8aa50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 -- 172.21.15.115:0/3449543895 shutdown_connections 2023-07-14T18:11:57.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 -- 172.21.15.115:0/3449543895 wait complete. 2023-07-14T18:11:57.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.001+0000 7f6e9139fb80 1 -- start start 2023-07-14T18:11:57.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e9139fb80 1 Processor -- start 2023-07-14T18:11:57.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ac3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e9139fb80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aac460 0x55bdb8ac6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb8ac7590 con 0x55bdb8aab860 2023-07-14T18:11:57.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb8ac7700 con 0x55bdb8aaaf10 2023-07-14T18:11:57.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e9139fb80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdb8ac78c0 con 0x55bdb8aac460 2023-07-14T18:11:57.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e9139fb80 10 client.0 _ll_register_callbacks cb 0x55bdb8999710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:57.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7e59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54182/0 (socket says 172.21.15.115:54182) 2023-07-14T18:11:57.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7e59c700 1 -- 172.21.15.115:0/349056917 learned_addr learned my addr 172.21.15.115:0/349056917 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.049 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7e59c700 1 -- 172.21.15.115:0/349056917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aac460 msgr2=0x55bdb8ac6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aac460 0x55bdb8ac6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7e59c700 1 -- 172.21.15.115:0/349056917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 msgr2=0x55bdb8ac3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ac3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ac3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7e59c700 1 -- 172.21.15.115:0/349056917 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdb8829d90 con 0x55bdb8aab860 2023-07-14T18:11:57.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ac3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:57.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e7e59c700 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac4110 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6e700a1d30 tx=0x7f6e70099d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e5be575cd343ff server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.050 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e667fc700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:57.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e667fc700 1 -- 172.21.15.115:0/349056917 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70096150 con 0x55bdb8aab860 2023-07-14T18:11:57.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e667fc700 1 -- 172.21.15.115:0/349056917 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6e7000f8e0 con 0x55bdb8aab860 2023-07-14T18:11:57.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.002+0000 7f6e9139fb80 1 -- 172.21.15.115:0/349056917 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e700a2260 con 0x55bdb8aab860 2023-07-14T18:11:57.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e667fc700 1 -- 172.21.15.115:0/349056917 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e7009de40 con 0x55bdb8aab860 2023-07-14T18:11:57.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e9139fb80 10 client.15399 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:57.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e9139fb80 1 -- 172.21.15.115:0/349056917 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bdb8850200 con 0x55bdb8aab860 2023-07-14T18:11:57.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e667fc700 1 -- 172.21.15.115:0/349056917 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6e700a2260 con 0x55bdb8aab860 2023-07-14T18:11:57.051 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e667fc700 1 -- 172.21.15.115:0/349056917 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6e70095c00 con 0x55bdb8aab860 2023-07-14T18:11:57.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e9139fb80 10 client.15399 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:57.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e9139fb80 10 client.15399 resolve_mds: resolved name 'f' to daemon mds.f 2023-07-14T18:11:57.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e9139fb80 20 client.15399 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:57.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55bdb8b18a10 0x55bdb8b15880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e9139fb80 4 client.15399 mds_command: new command op to 24463 tid=0[{"prefix": "status"}] 2023-07-14T18:11:57.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e9139fb80 1 -- 172.21.15.115:0/349056917 --> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bdb89f4e60 con 0x55bdb8b18a10 2023-07-14T18:11:57.052 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.003+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55bdb8b18a10 0x55bdb8b15880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e667fc700 10 client.15399 ms_handle_connect on v2:172.21.15.187:6834/4258537735 2023-07-14T18:11:57.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e7ed9d700 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55bdb8b18a10 0x55bdb8b15880 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ca91fd9a42fc7cca server_cookie=dba0ec63557eb109 in_seq=0 out_seq=0 2023-07-14T18:11:57.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e667fc700 1 -- 172.21.15.115:0/349056917 <== mds.2 v2:172.21.15.187:6834/4258537735 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55bdb89f4e60 con 0x55bdb8b18a10 2023-07-14T18:11:57.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e667fc700 10 client.15399 handle_command_reply: tid=0 2023-07-14T18:11:57.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 client.15399 shutdown 2023-07-14T18:11:57.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 -- 172.21.15.115:0/349056917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 msgr2=0x55bdb8ac4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac4110 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6e700a1d30 tx=0x7f6e70099d10 comp rx=0 tx=0).stop 2023-07-14T18:11:57.053 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 -- 172.21.15.115:0/349056917 shutdown_connections 2023-07-14T18:11:57.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bdb8aac460 0x55bdb8ac6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bdb8aaaf10 0x55bdb8ac3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.187:6834/4258537735,v1:172.21.15.187:6835/4258537735] conn(0x55bdb8b18a10 0x55bdb8b15880 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 --2- 172.21.15.115:0/349056917 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bdb8aab860 0x55bdb8ac4110 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e667fc700 0 client.15399 ms_handle_reset on v2:172.21.15.187:6834/4258537735 2023-07-14T18:11:57.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 -- 172.21.15.115:0/349056917 >> 172.21.15.115:0/349056917 conn(0x55bdb8aa0630 msgr2=0x55bdb8ab2810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 -- 172.21.15.115:0/349056917 shutdown_connections 2023-07-14T18:11:57.054 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 1 -- 172.21.15.115:0/349056917 wait complete. 2023-07-14T18:11:57.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.004+0000 7f6e9139fb80 20 client.15399 trim_cache size 0 max 16384 2023-07-14T18:11:57.055 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:11:57.055 INFO:teuthology.orchestra.run.smithi115.stdout: "cluster_fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:11:57.055 INFO:teuthology.orchestra.run.smithi115.stdout: "whoami": 2, 2023-07-14T18:11:57.055 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 24463, 2023-07-14T18:11:57.056 INFO:teuthology.orchestra.run.smithi115.stdout: "want_state": "up:active", 2023-07-14T18:11:57.056 INFO:teuthology.orchestra.run.smithi115.stdout: "state": "up:active", 2023-07-14T18:11:57.056 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs", 2023-07-14T18:11:57.056 INFO:teuthology.orchestra.run.smithi115.stdout: "rank_uptime": 79.585660078000004, 2023-07-14T18:11:57.056 INFO:teuthology.orchestra.run.smithi115.stdout: "mdsmap_epoch": 24, 2023-07-14T18:11:57.056 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch": 89, 2023-07-14T18:11:57.056 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch_barrier": 87, 2023-07-14T18:11:57.056 INFO:teuthology.orchestra.run.smithi115.stdout: "uptime": 118.632578861 2023-07-14T18:11:57.057 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:11:57.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 -- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6e6c0a11e0 msgr2=0x7f6e6c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6e6c0a11e0 0x7f6e6c0a3690 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6e800ffbc0 tx=0x7f6e68008040 comp rx=0 tx=0).stop 2023-07-14T18:11:57.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 -- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e800ffd60 msgr2=0x7f6e8019a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e800ffd60 0x7f6e8019a6b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000c9b0 tx=0x7f6e7000cd70 comp rx=0 tx=0).stop 2023-07-14T18:11:57.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 -- 172.21.15.115:0/3395429664 shutdown_connections 2023-07-14T18:11:57.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f6e6c0a11e0 0x7f6e6c0a3690 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.058 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6e801006f0 0x7f6e8019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e800feb70 0x7f6e80069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 --2- 172.21.15.115:0/3395429664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6e800ffd60 0x7f6e8019a6b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 -- 172.21.15.115:0/3395429664 >> 172.21.15.115:0/3395429664 conn(0x7f6e800fa1d0 msgr2=0x7f6e800fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 -- 172.21.15.115:0/3395429664 shutdown_connections 2023-07-14T18:11:57.059 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.005+0000 7f6e85207700 1 -- 172.21.15.115:0/3395429664 wait complete. 2023-07-14T18:11:57.060 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T18:11:57.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 -- 172.21.15.115:0/451316646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f14100690 msgr2=0x7f9f14106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 --2- 172.21.15.115:0/451316646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f14100690 0x7f9f14106f50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00b0a0 tx=0x7f9f0c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 -- 172.21.15.115:0/451316646 shutdown_connections 2023-07-14T18:11:57.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 --2- 172.21.15.115:0/451316646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f14100690 0x7f9f14106f50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 --2- 172.21.15.115:0/451316646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f140feb10 0x7f9f140fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 --2- 172.21.15.115:0/451316646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f140ffd00 0x7f9f14100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 -- 172.21.15.115:0/451316646 >> 172.21.15.115:0/451316646 conn(0x7f9f140fa130 msgr2=0x7f9f140fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 -- 172.21.15.115:0/451316646 shutdown_connections 2023-07-14T18:11:57.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 -- 172.21.15.115:0/451316646 wait complete. 2023-07-14T18:11:57.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 Processor -- start 2023-07-14T18:11:57.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.142+0000 7f9f19770700 1 -- start start 2023-07-14T18:11:57.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f19770700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f140feb10 0x7f9f14195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f19770700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f140ffd00 0x7f9f14195f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f19770700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f14100690 0x7f9f1419cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f19770700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1419d580 con 0x7f9f140feb10 2023-07-14T18:11:57.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f19770700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1419d6c0 con 0x7f9f14100690 2023-07-14T18:11:57.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f19770700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1419d850 con 0x7f9f140ffd00 2023-07-14T18:11:57.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f12ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f140feb10 0x7f9f14195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.186 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f12ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f140feb10 0x7f9f14195a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54194/0 (socket says 172.21.15.115:54194) 2023-07-14T18:11:57.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f12ffd700 1 -- 172.21.15.115:0/4058620760 learned_addr learned my addr 172.21.15.115:0/4058620760 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f137fe700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f14100690 0x7f9f1419cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f127fc700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f140ffd00 0x7f9f14195f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f12ffd700 1 -- 172.21.15.115:0/4058620760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f140ffd00 msgr2=0x7f9f14195f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.187 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f140ffd00 0x7f9f14195f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.189 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f12ffd700 1 -- 172.21.15.115:0/4058620760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f14100690 msgr2=0x7f9f1419cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f14100690 0x7f9f1419cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f12ffd700 1 -- 172.21.15.115:0/4058620760 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c009d00 con 0x7f9f140feb10 2023-07-14T18:11:57.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f140feb10 0x7f9f14195a00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400d980 tx=0x7f9f0400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9efbfff700 1 -- 172.21.15.115:0/4058620760 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0400de60 con 0x7f9f140feb10 2023-07-14T18:11:57.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9efbfff700 1 -- 172.21.15.115:0/4058620760 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9f040044b0 con 0x7f9f140feb10 2023-07-14T18:11:57.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f19770700 1 -- 172.21.15.115:0/4058620760 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1419db30 con 0x7f9f140feb10 2023-07-14T18:11:57.190 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9efbfff700 1 -- 172.21.15.115:0/4058620760 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04003b40 con 0x7f9f140feb10 2023-07-14T18:11:57.191 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.143+0000 7f9f19770700 1 -- 172.21.15.115:0/4058620760 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1419e040 con 0x7f9f140feb10 2023-07-14T18:11:57.195 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.146+0000 7f9efbfff700 1 -- 172.21.15.115:0/4058620760 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f9f04003ca0 con 0x7f9f140feb10 2023-07-14T18:11:57.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f25908b80 1 Processor -- start 2023-07-14T18:11:57.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9efbfff700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9efc0a11e0 0x7f9efc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f25908b80 1 -- start start 2023-07-14T18:11:57.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9efbfff700 1 -- 172.21.15.115:0/4058620760 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9f040ccae0 con 0x7f9f140feb10 2023-07-14T18:11:57.196 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f127fc700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9efc0a11e0 0x7f9efc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32958f10 0x55bc32959320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32959860 0x55bc32959e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc3295a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f25908b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc3285dc60 con 0x55bc32958f10 2023-07-14T18:11:57.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f25908b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc328b09f0 con 0x55bc32959860 2023-07-14T18:11:57.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f25908b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc328a9e80 con 0x55bc3295a460 2023-07-14T18:11:57.197 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f12ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32958f10 0x55bc32959320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f12ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32958f10 0x55bc32959320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54202/0 (socket says 172.21.15.115:54202) 2023-07-14T18:11:57.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f12ffd700 1 -- 172.21.15.115:0/3264996405 learned_addr learned my addr 172.21.15.115:0/3264996405 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f127fc700 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32959860 0x55bc32959e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f137fe700 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc3295a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f12ffd700 1 -- 172.21.15.115:0/3264996405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 msgr2=0x55bc3295a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc3295a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.198 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f127fc700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9efc0a11e0 0x7f9efc0a3690 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009c20 tx=0x7f9f080113e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f12ffd700 1 -- 172.21.15.115:0/3264996405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32959860 msgr2=0x55bc32959e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32959860 0x55bc32959e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.147+0000 7f9f12ffd700 1 -- 172.21.15.115:0/3264996405 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc326d7d90 con 0x55bc32958f10 2023-07-14T18:11:57.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.148+0000 7f9f137fe700 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc3295a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:57.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.148+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32958f10 0x55bc32959320 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9f040a19a0 tx=0x7f9f040993e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30c154e506adad server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.148+0000 7f9ef97fa700 1 -- 172.21.15.115:0/3264996405 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04009bb0 con 0x55bc32958f10 2023-07-14T18:11:57.199 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.148+0000 7f9ef97fa700 1 -- 172.21.15.115:0/3264996405 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9f04099a80 con 0x55bc32958f10 2023-07-14T18:11:57.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.148+0000 7f9ef97fa700 1 -- 172.21.15.115:0/3264996405 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f040a1a40 con 0x55bc32958f10 2023-07-14T18:11:57.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.148+0000 7f9f25908b80 1 -- 172.21.15.115:0/3264996405 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32958f10 msgr2=0x55bc32959320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.148+0000 7f9f25908b80 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32958f10 0x55bc32959320 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9f040a19a0 tx=0x7f9f040993e0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 -- 172.21.15.115:0/3264996405 shutdown_connections 2023-07-14T18:11:57.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc3295a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32958f10 0x55bc32959320 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.200 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 --2- 172.21.15.115:0/3264996405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32959860 0x55bc32959e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 -- 172.21.15.115:0/3264996405 >> 172.21.15.115:0/3264996405 conn(0x55bc3294e630 msgr2=0x55bc32953980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 -- 172.21.15.115:0/3264996405 shutdown_connections 2023-07-14T18:11:57.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 -- 172.21.15.115:0/3264996405 wait complete. 2023-07-14T18:11:57.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 -- start start 2023-07-14T18:11:57.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 Processor -- start 2023-07-14T18:11:57.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32962810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.201 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32962d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc32963d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc32849a70 con 0x55bc32959860 2023-07-14T18:11:57.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc32849770 con 0x55bc32958f10 2023-07-14T18:11:57.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc3272d8d0 con 0x55bc3295a460 2023-07-14T18:11:57.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f25908b80 10 client.0 _ll_register_callbacks cb 0x55bc32847710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:57.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f127fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32962d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f127fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32962d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54210/0 (socket says 172.21.15.115:54210) 2023-07-14T18:11:57.202 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f127fc700 1 -- 172.21.15.115:0/545442924 learned_addr learned my addr 172.21.15.115:0/545442924 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32962810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.149+0000 7f9f137fe700 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc32963d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9f127fc700 1 -- 172.21.15.115:0/545442924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 msgr2=0x55bc32963d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9f127fc700 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc32963d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9f127fc700 1 -- 172.21.15.115:0/545442924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 msgr2=0x55bc32962810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9f127fc700 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32962810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.203 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9f127fc700 1 -- 172.21.15.115:0/545442924 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1419e040 con 0x55bc32959860 2023-07-14T18:11:57.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9f127fc700 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32962d50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9f08014920 tx=0x7f9f080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c1a4af8d40219e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9ef6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:57.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/545442924 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08020730 con 0x55bc32959860 2023-07-14T18:11:57.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/545442924 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9f0801a040 con 0x55bc32959860 2023-07-14T18:11:57.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9f25908b80 1 -- 172.21.15.115:0/545442924 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc328034a0 con 0x55bc32959860 2023-07-14T18:11:57.204 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/545442924 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f080142e0 con 0x55bc32959860 2023-07-14T18:11:57.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9f25908b80 10 client.15411 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:57.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9f25908b80 1 -- 172.21.15.115:0/545442924 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bc326c0c10 con 0x55bc32959860 2023-07-14T18:11:57.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/545442924 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55bc328034a0 con 0x55bc32959860 2023-07-14T18:11:57.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.150+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/545442924 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9f08012950 con 0x55bc32959860 2023-07-14T18:11:57.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.151+0000 7f9f25908b80 10 client.15411 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:57.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.151+0000 7f9f25908b80 10 client.15411 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T18:11:57.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.151+0000 7f9f25908b80 20 client.15411 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:57.205 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.151+0000 7f9f25908b80 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x55bc32976380 0x55bc32975760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.151+0000 7f9f25908b80 4 client.15411 mds_command: new command op to 24493 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:11:57.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.151+0000 7f9f25908b80 1 -- 172.21.15.115:0/545442924 --> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc32927d50 con 0x55bc32976380 2023-07-14T18:11:57.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.151+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x55bc32976380 0x55bc32975760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.151+0000 7f9ef6ff5700 10 client.15411 ms_handle_connect on v2:172.21.15.187:6838/1449243194 2023-07-14T18:11:57.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.151+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x55bc32976380 0x55bc32975760 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ef60fc9dd354fcd server_cookie=b9bd371183e3f39 in_seq=0 out_seq=0 2023-07-14T18:11:57.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/545442924 <== mds.1 v2:172.21.15.187:6838/1449243194 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55bc32927d50 con 0x55bc32976380 2023-07-14T18:11:57.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9ef6ff5700 10 client.15411 handle_command_reply: tid=0 2023-07-14T18:11:57.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 client.15411 shutdown 2023-07-14T18:11:57.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 -- 172.21.15.115:0/545442924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 msgr2=0x55bc32962d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32962d50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9f08014920 tx=0x7f9f080047c0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 -- 172.21.15.115:0/545442924 shutdown_connections 2023-07-14T18:11:57.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x55bc32976380 0x55bc32975760 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc32963d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32962810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9ef6ff5700 0 client.15411 ms_handle_reset on v2:172.21.15.187:6838/1449243194 2023-07-14T18:11:57.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 --2- 172.21.15.115:0/545442924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32962d50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 -- 172.21.15.115:0/545442924 >> 172.21.15.115:0/545442924 conn(0x55bc3294e630 msgr2=0x55bc32952600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 -- 172.21.15.115:0/545442924 shutdown_connections 2023-07-14T18:11:57.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 1 -- 172.21.15.115:0/545442924 wait complete. 2023-07-14T18:11:57.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.152+0000 7f9f25908b80 20 client.15411 trim_cache size 0 max 16384 2023-07-14T18:11:57.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.169+0000 7f9f25908b80 1 Processor -- start 2023-07-14T18:11:57.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.169+0000 7f9f25908b80 1 -- start start 2023-07-14T18:11:57.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.169+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32975760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.169+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32975ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.169+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc3295d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.169+0000 7f9f25908b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc32878790 con 0x55bc32959860 2023-07-14T18:11:57.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.169+0000 7f9f25908b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc32739de0 con 0x55bc32958f10 2023-07-14T18:11:57.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.169+0000 7f9f25908b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc32868480 con 0x55bc3295a460 2023-07-14T18:11:57.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f127fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32975ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f127fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32975ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54218/0 (socket says 172.21.15.115:54218) 2023-07-14T18:11:57.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f127fc700 1 -- 172.21.15.115:0/3412120350 learned_addr learned my addr 172.21.15.115:0/3412120350 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f127fc700 1 -- 172.21.15.115:0/3412120350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 msgr2=0x55bc3295d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f127fc700 1 --2- 172.21.15.115:0/3412120350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc3295d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f127fc700 1 -- 172.21.15.115:0/3412120350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 msgr2=0x55bc32975760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/3412120350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32975760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f127fc700 1 --2- 172.21.15.115:0/3412120350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32975760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f127fc700 1 -- 172.21.15.115:0/3412120350 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc32953f00 con 0x55bc32959860 2023-07-14T18:11:57.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9f127fc700 1 --2- 172.21.15.115:0/3412120350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32975ca0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009820 tx=0x7f9f08005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f0d2b991759732 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9ef9ffb700 1 -- 172.21.15.115:0/3412120350 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08004220 con 0x55bc32959860 2023-07-14T18:11:57.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9ef9ffb700 1 -- 172.21.15.115:0/3412120350 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9f080142e0 con 0x55bc32959860 2023-07-14T18:11:57.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.170+0000 7f9ef9ffb700 1 -- 172.21.15.115:0/3412120350 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f080039f0 con 0x55bc32959860 2023-07-14T18:11:57.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 -- 172.21.15.115:0/3412120350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 msgr2=0x55bc32975ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 --2- 172.21.15.115:0/3412120350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32975ca0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009820 tx=0x7f9f08005a20 comp rx=0 tx=0).stop 2023-07-14T18:11:57.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 -- 172.21.15.115:0/3412120350 shutdown_connections 2023-07-14T18:11:57.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 --2- 172.21.15.115:0/3412120350 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc3295a460 0x55bc3295d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 --2- 172.21.15.115:0/3412120350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32975760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 --2- 172.21.15.115:0/3412120350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc32959860 0x55bc32975ca0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.217 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 -- 172.21.15.115:0/3412120350 >> 172.21.15.115:0/3412120350 conn(0x55bc3294e630 msgr2=0x55bc329530a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 -- 172.21.15.115:0/3412120350 shutdown_connections 2023-07-14T18:11:57.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 -- 172.21.15.115:0/3412120350 wait complete. 2023-07-14T18:11:57.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 -- start start 2023-07-14T18:11:57.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 Processor -- start 2023-07-14T18:11:57.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32971e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc32959860 0x55bc32972390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc3295a460 0x55bc329728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc32972e10 con 0x55bc3295a460 2023-07-14T18:11:57.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc32972f80 con 0x55bc32958f10 2023-07-14T18:11:57.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc32973140 con 0x55bc32959860 2023-07-14T18:11:57.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f25908b80 10 client.0 _ll_register_callbacks cb 0x55bc32847710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:57.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f137fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc3295a460 0x55bc329728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f137fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc3295a460 0x55bc329728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54226/0 (socket says 172.21.15.115:54226) 2023-07-14T18:11:57.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f137fe700 1 -- 172.21.15.115:0/1279546725 learned_addr learned my addr 172.21.15.115:0/1279546725 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f127fc700 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc32959860 0x55bc32972390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.171+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32971e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f137fe700 1 -- 172.21.15.115:0/1279546725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc32959860 msgr2=0x55bc32972390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f137fe700 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc32959860 0x55bc32972390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f137fe700 1 -- 172.21.15.115:0/1279546725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 msgr2=0x55bc32971e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f137fe700 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32971e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f137fe700 1 -- 172.21.15.115:0/1279546725 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc326c0c10 con 0x55bc3295a460 2023-07-14T18:11:57.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32971e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:11:57.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f137fe700 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc3295a460 0x55bc329728d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c007950 tx=0x7f9f0c007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73470f9eafbb18 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9ef6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:57.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/1279546725 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c00e040 con 0x55bc3295a460 2023-07-14T18:11:57.221 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/1279546725 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9f0c004a50 con 0x55bc3295a460 2023-07-14T18:11:57.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f25908b80 1 -- 172.21.15.115:0/1279546725 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f04099a80 con 0x55bc3295a460 2023-07-14T18:11:57.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/1279546725 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f0c009b80 con 0x55bc3295a460 2023-07-14T18:11:57.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f25908b80 10 client.15417 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:57.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9f25908b80 1 -- 172.21.15.115:0/1279546725 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bc326fe200 con 0x55bc3295a460 2023-07-14T18:11:57.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/1279546725 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9f04099a80 con 0x55bc3295a460 2023-07-14T18:11:57.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.172+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/1279546725 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9f0c00b0d0 con 0x55bc3295a460 2023-07-14T18:11:57.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.173+0000 7f9f25908b80 10 client.15417 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:57.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.173+0000 7f9f25908b80 10 client.15417 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T18:11:57.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.173+0000 7f9f25908b80 20 client.15417 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:57.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.173+0000 7f9f25908b80 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x55bc329d0050 0x55bc329cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.173+0000 7f9f25908b80 4 client.15417 mds_command: new command op to 24493 tid=0[{"prefix": "status"}] 2023-07-14T18:11:57.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.173+0000 7f9f25908b80 1 -- 172.21.15.115:0/1279546725 --> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bc328a2e60 con 0x55bc329d0050 2023-07-14T18:11:57.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.173+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x55bc329d0050 0x55bc329cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.175+0000 7f9ef6ff5700 10 client.15417 ms_handle_connect on v2:172.21.15.187:6838/1449243194 2023-07-14T18:11:57.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.175+0000 7f9f12ffd700 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x55bc329d0050 0x55bc329cc9c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2255ea1763a0162a server_cookie=480e843dc1df02c9 in_seq=0 out_seq=0 2023-07-14T18:11:57.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9ef6ff5700 1 -- 172.21.15.115:0/1279546725 <== mds.1 v2:172.21.15.187:6838/1449243194 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55bc328a2e60 con 0x55bc329d0050 2023-07-14T18:11:57.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9ef6ff5700 10 client.15417 handle_command_reply: tid=0 2023-07-14T18:11:57.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 client.15417 shutdown 2023-07-14T18:11:57.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 -- 172.21.15.115:0/1279546725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc3295a460 msgr2=0x55bc329728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc3295a460 0x55bc329728d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c007950 tx=0x7f9f0c007d50 comp rx=0 tx=0).stop 2023-07-14T18:11:57.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 -- 172.21.15.115:0/1279546725 shutdown_connections 2023-07-14T18:11:57.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.187:6838/1449243194,v1:172.21.15.187:6839/1449243194] conn(0x55bc329d0050 0x55bc329cc9c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55bc3295a460 0x55bc329728d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9ef6ff5700 0 client.15417 ms_handle_reset on v2:172.21.15.187:6838/1449243194 2023-07-14T18:11:57.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bc32958f10 0x55bc32971e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 --2- 172.21.15.115:0/1279546725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55bc32959860 0x55bc32972390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 -- 172.21.15.115:0/1279546725 >> 172.21.15.115:0/1279546725 conn(0x55bc3294e630 msgr2=0x55bc32975760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 -- 172.21.15.115:0/1279546725 shutdown_connections 2023-07-14T18:11:57.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 1 -- 172.21.15.115:0/1279546725 wait complete. 2023-07-14T18:11:57.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f25908b80 20 client.15417 trim_cache size 0 max 16384 2023-07-14T18:11:57.226 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:11:57.226 INFO:teuthology.orchestra.run.smithi115.stdout: "cluster_fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:11:57.226 INFO:teuthology.orchestra.run.smithi115.stdout: "whoami": 1, 2023-07-14T18:11:57.227 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 24493, 2023-07-14T18:11:57.227 INFO:teuthology.orchestra.run.smithi115.stdout: "want_state": "up:active", 2023-07-14T18:11:57.227 INFO:teuthology.orchestra.run.smithi115.stdout: "state": "up:active", 2023-07-14T18:11:57.227 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs", 2023-07-14T18:11:57.227 INFO:teuthology.orchestra.run.smithi115.stdout: "rank_uptime": 80.205031433000002, 2023-07-14T18:11:57.227 INFO:teuthology.orchestra.run.smithi115.stdout: "mdsmap_epoch": 24, 2023-07-14T18:11:57.227 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch": 89, 2023-07-14T18:11:57.228 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch_barrier": 87, 2023-07-14T18:11:57.228 INFO:teuthology.orchestra.run.smithi115.stdout: "uptime": 112.397778863 2023-07-14T18:11:57.228 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:11:57.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 -- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9efc0a11e0 msgr2=0x7f9efc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9efc0a11e0 0x7f9efc0a3690 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009c20 tx=0x7f9f080113e0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 -- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f140feb10 msgr2=0x7f9f14195a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f140feb10 0x7f9f14195a00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400d980 tx=0x7f9f0400dd40 comp rx=0 tx=0).stop 2023-07-14T18:11:57.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 -- 172.21.15.115:0/4058620760 shutdown_connections 2023-07-14T18:11:57.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f9efc0a11e0 0x7f9efc0a3690 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f14100690 0x7f9f1419cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9f140feb10 0x7f9f14195a00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 --2- 172.21.15.115:0/4058620760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f140ffd00 0x7f9f14195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.176+0000 7f9f19770700 1 -- 172.21.15.115:0/4058620760 >> 172.21.15.115:0/4058620760 conn(0x7f9f140fa130 msgr2=0x7f9f140677a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.230 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.177+0000 7f9f19770700 1 -- 172.21.15.115:0/4058620760 shutdown_connections 2023-07-14T18:11:57.231 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.177+0000 7f9f19770700 1 -- 172.21.15.115:0/4058620760 wait complete. 2023-07-14T18:11:57.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T18:11:57.353 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.313+0000 7f493bead700 1 -- 172.21.15.115:0/4146448315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 msgr2=0x7f4934105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.313+0000 7f493bead700 1 --2- 172.21.15.115:0/4146448315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 0x7f4934105390 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4928009990 tx=0x7f4928009ca0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.313+0000 7f493bead700 1 -- 172.21.15.115:0/4146448315 shutdown_connections 2023-07-14T18:11:57.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.313+0000 7f493bead700 1 --2- 172.21.15.115:0/4146448315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49341058d0 0x7f4934107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.313+0000 7f493bead700 1 --2- 172.21.15.115:0/4146448315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4934100690 0x7f4934102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.313+0000 7f493bead700 1 --2- 172.21.15.115:0/4146448315 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 0x7f4934105390 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.354 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.313+0000 7f493bead700 1 -- 172.21.15.115:0/4146448315 >> 172.21.15.115:0/4146448315 conn(0x7f49340fa130 msgr2=0x7f49340fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.313+0000 7f493bead700 1 -- 172.21.15.115:0/4146448315 shutdown_connections 2023-07-14T18:11:57.355 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.313+0000 7f493bead700 1 -- 172.21.15.115:0/4146448315 wait complete. 2023-07-14T18:11:57.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f493bead700 1 Processor -- start 2023-07-14T18:11:57.356 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f493bead700 1 -- start start 2023-07-14T18:11:57.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f493bead700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4934100690 0x7f4934100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f493bead700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 0x7f49340fe640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f493bead700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49341058d0 0x7f49340feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f493bead700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49340ff320 con 0x7f4934102fb0 2023-07-14T18:11:57.357 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f493bead700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49340ff490 con 0x7f49341058d0 2023-07-14T18:11:57.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f493bead700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493419f8a0 con 0x7f4934100690 2023-07-14T18:11:57.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939448700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 0x7f49340fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939448700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 0x7f49340fe640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54238/0 (socket says 172.21.15.115:54238) 2023-07-14T18:11:57.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939448700 1 -- 172.21.15.115:0/2047333087 learned_addr learned my addr 172.21.15.115:0/2047333087 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939c49700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4934100690 0x7f4934100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.358 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939448700 1 -- 172.21.15.115:0/2047333087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4934100690 msgr2=0x7f4934100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939448700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4934100690 0x7f4934100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939448700 1 -- 172.21.15.115:0/2047333087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49341058d0 msgr2=0x7f49340feb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939448700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49341058d0 0x7f49340feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939448700 1 -- 172.21.15.115:0/2047333087 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49280095f0 con 0x7f4934102fb0 2023-07-14T18:11:57.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4939448700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 0x7f49340fe640 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4928006010 tx=0x7f4928004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4932ffd700 1 -- 172.21.15.115:0/2047333087 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492801d070 con 0x7f4934102fb0 2023-07-14T18:11:57.359 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4932ffd700 1 -- 172.21.15.115:0/2047333087 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f49280041b0 con 0x7f4934102fb0 2023-07-14T18:11:57.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f493bead700 1 -- 172.21.15.115:0/2047333087 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493419fb20 con 0x7f4934102fb0 2023-07-14T18:11:57.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.314+0000 7f4932ffd700 1 -- 172.21.15.115:0/2047333087 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928022da0 con 0x7f4934102fb0 2023-07-14T18:11:57.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.315+0000 7f493bead700 1 -- 172.21.15.115:0/2047333087 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493419ffa0 con 0x7f4934102fb0 2023-07-14T18:11:57.360 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.316+0000 7f4932ffd700 1 -- 172.21.15.115:0/2047333087 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f4928022570 con 0x7f4934102fb0 2023-07-14T18:11:57.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.317+0000 7f4932ffd700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f491c0a1260 0x7f491c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.317+0000 7f4939c49700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f491c0a1260 0x7f491c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.317+0000 7f4932ffd700 1 -- 172.21.15.115:0/2047333087 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f49280ce260 con 0x7f4934102fb0 2023-07-14T18:11:57.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.317+0000 7f4939c49700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f491c0a1260 0x7f491c0a3710 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f492c009d10 tx=0x7f492c009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.365 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.318+0000 7f4948045b80 1 Processor -- start 2023-07-14T18:11:57.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.318+0000 7f4948045b80 1 -- start start 2023-07-14T18:11:57.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.318+0000 7f4948045b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 0x560148f75320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.318+0000 7f4948045b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f75860 0x560148f75e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.318+0000 7f4948045b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f768d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.318+0000 7f4948045b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148e79c60 con 0x560148f76460 2023-07-14T18:11:57.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.318+0000 7f4948045b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148ecc9f0 con 0x560148f75860 2023-07-14T18:11:57.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.318+0000 7f4948045b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148ec5e80 con 0x560148f74f10 2023-07-14T18:11:57.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f493a44a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f493a44a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f768d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54248/0 (socket says 172.21.15.115:54248) 2023-07-14T18:11:57.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f493a44a700 1 -- 172.21.15.115:0/1736811368 learned_addr learned my addr 172.21.15.115:0/1736811368 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f4939c49700 1 --2- 172.21.15.115:0/1736811368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 0x560148f75320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f4939448700 1 --2- 172.21.15.115:0/1736811368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f75860 0x560148f75e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f493a44a700 1 -- 172.21.15.115:0/1736811368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 msgr2=0x560148f75320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.367 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f493a44a700 1 --2- 172.21.15.115:0/1736811368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 0x560148f75320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f493a44a700 1 -- 172.21.15.115:0/1736811368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f75860 msgr2=0x560148f75e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f493a44a700 1 --2- 172.21.15.115:0/1736811368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f75860 0x560148f75e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f493a44a700 1 -- 172.21.15.115:0/1736811368 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560148cf3d90 con 0x560148f76460 2023-07-14T18:11:57.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f493a44a700 1 --2- 172.21.15.115:0/1736811368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f768d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f492400bae0 tx=0x7f492400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7866cdb569837b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f491bfff700 1 -- 172.21.15.115:0/1736811368 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492400d890 con 0x560148f76460 2023-07-14T18:11:57.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f491bfff700 1 -- 172.21.15.115:0/1736811368 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f492400de80 con 0x560148f76460 2023-07-14T18:11:57.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.319+0000 7f491bfff700 1 -- 172.21.15.115:0/1736811368 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924005390 con 0x560148f76460 2023-07-14T18:11:57.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 -- 172.21.15.115:0/1736811368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 msgr2=0x560148f768d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 --2- 172.21.15.115:0/1736811368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f768d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f492400bae0 tx=0x7f492400bdf0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 -- 172.21.15.115:0/1736811368 shutdown_connections 2023-07-14T18:11:57.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 --2- 172.21.15.115:0/1736811368 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f768d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 --2- 172.21.15.115:0/1736811368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 0x560148f75320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 --2- 172.21.15.115:0/1736811368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f75860 0x560148f75e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 -- 172.21.15.115:0/1736811368 >> 172.21.15.115:0/1736811368 conn(0x560148f6a630 msgr2=0x560148f6f980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 -- 172.21.15.115:0/1736811368 shutdown_connections 2023-07-14T18:11:57.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 -- 172.21.15.115:0/1736811368 wait complete. 2023-07-14T18:11:57.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 -- start start 2023-07-14T18:11:57.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.320+0000 7f4948045b80 1 Processor -- start 2023-07-14T18:11:57.370 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f4948045b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 0x560148f7e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f4948045b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f75860 0x560148f7ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f4948045b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f85e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f4948045b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148e65a70 con 0x560148f76460 2023-07-14T18:11:57.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f4948045b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148e65770 con 0x560148f74f10 2023-07-14T18:11:57.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f4948045b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148d498d0 con 0x560148f75860 2023-07-14T18:11:57.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f4948045b80 10 client.0 _ll_register_callbacks cb 0x560148e63710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:57.371 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f493a44a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f85e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f493a44a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f85e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54254/0 (socket says 172.21.15.115:54254) 2023-07-14T18:11:57.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f493a44a700 1 -- 172.21.15.115:0/1258193509 learned_addr learned my addr 172.21.15.115:0/1258193509 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f4939c49700 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 0x560148f7e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f4939448700 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f75860 0x560148f7ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f493a44a700 1 -- 172.21.15.115:0/1258193509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f75860 msgr2=0x560148f7ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f493a44a700 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f75860 0x560148f7ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f493a44a700 1 -- 172.21.15.115:0/1258193509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 msgr2=0x560148f7e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.372 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f493a44a700 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 0x560148f7e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.321+0000 7f493a44a700 1 -- 172.21.15.115:0/1258193509 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492400b740 con 0x560148f76460 2023-07-14T18:11:57.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f493a44a700 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f85e40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4924004b80 tx=0x7f4924004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a357303b54e4d6ca server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f49197fa700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:57.373 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f49197fa700 1 -- 172.21.15.115:0/1258193509 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492401e070 con 0x560148f76460 2023-07-14T18:11:57.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f49197fa700 1 -- 172.21.15.115:0/1258193509 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4924004320 con 0x560148f76460 2023-07-14T18:11:57.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4948045b80 1 -- 172.21.15.115:0/1258193509 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560148f43d50 con 0x560148f76460 2023-07-14T18:11:57.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f49197fa700 1 -- 172.21.15.115:0/1258193509 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49240044e0 con 0x560148f76460 2023-07-14T18:11:57.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4948045b80 10 client.15429 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:57.374 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4948045b80 1 -- 172.21.15.115:0/1258193509 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560148cdcc10 con 0x560148f76460 2023-07-14T18:11:57.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f49197fa700 1 -- 172.21.15.115:0/1258193509 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x560148f43d50 con 0x560148f76460 2023-07-14T18:11:57.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f49197fa700 1 -- 172.21.15.115:0/1258193509 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4924003680 con 0x560148f76460 2023-07-14T18:11:57.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4948045b80 10 client.15429 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:57.375 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4948045b80 10 client.15429 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:11:57.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4948045b80 20 client.15429 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:57.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4948045b80 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x560148f89a60 0x560148f88ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4948045b80 4 client.15429 mds_command: new command op to 24515 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:11:57.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4948045b80 1 -- 172.21.15.115:0/1258193509 --> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560148ebee60 con 0x560148f89a60 2023-07-14T18:11:57.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.322+0000 7f4939c49700 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x560148f89a60 0x560148f88ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.323+0000 7f49197fa700 10 client.15429 ms_handle_connect on v2:172.21.15.150:6838/2895627668 2023-07-14T18:11:57.376 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.323+0000 7f4939c49700 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x560148f89a60 0x560148f88ee0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23e5d535fc8aebbb server_cookie=9fe7f323ea7a23f9 in_seq=0 out_seq=0 2023-07-14T18:11:57.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.323+0000 7f49197fa700 1 -- 172.21.15.115:0/1258193509 <== mds.0 v2:172.21.15.150:6838/2895627668 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x560148ebee60 con 0x560148f89a60 2023-07-14T18:11:57.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.323+0000 7f49197fa700 10 client.15429 handle_command_reply: tid=0 2023-07-14T18:11:57.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.323+0000 7f4948045b80 1 client.15429 shutdown 2023-07-14T18:11:57.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 -- 172.21.15.115:0/1258193509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 msgr2=0x560148f85e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f85e40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4924004b80 tx=0x7f4924004bb0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 -- 172.21.15.115:0/1258193509 shutdown_connections 2023-07-14T18:11:57.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x560148f89a60 0x560148f88ee0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.377 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f76460 0x560148f85e40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f49197fa700 0 client.15429 ms_handle_reset on v2:172.21.15.150:6838/2895627668 2023-07-14T18:11:57.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 0x560148f7e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 --2- 172.21.15.115:0/1258193509 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f75860 0x560148f7ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 -- 172.21.15.115:0/1258193509 >> 172.21.15.115:0/1258193509 conn(0x560148f6a630 msgr2=0x560148f6e600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 -- 172.21.15.115:0/1258193509 shutdown_connections 2023-07-14T18:11:57.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 1 -- 172.21.15.115:0/1258193509 wait complete. 2023-07-14T18:11:57.379 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.324+0000 7f4948045b80 20 client.15429 trim_cache size 0 max 16384 2023-07-14T18:11:57.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4948045b80 1 Processor -- start 2023-07-14T18:11:57.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4948045b80 1 -- start start 2023-07-14T18:11:57.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4948045b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 0x560148f6f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4948045b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f927b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.383 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4948045b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f76460 0x560148f88ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4948045b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148f7fab0 con 0x560148f75860 2023-07-14T18:11:57.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4948045b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148e94790 con 0x560148f76460 2023-07-14T18:11:57.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4948045b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148e84480 con 0x560148f74f10 2023-07-14T18:11:57.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4939448700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f927b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4939448700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f927b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54260/0 (socket says 172.21.15.115:54260) 2023-07-14T18:11:57.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.341+0000 7f4939448700 1 -- 172.21.15.115:0/371879590 learned_addr learned my addr 172.21.15.115:0/371879590 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.384 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4939c49700 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 0x560148f6f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4939448700 1 -- 172.21.15.115:0/371879590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 msgr2=0x560148f6f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4939448700 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 0x560148f6f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f493a44a700 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f76460 0x560148f88ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4939448700 1 -- 172.21.15.115:0/371879590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f76460 msgr2=0x560148f88ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.385 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4939448700 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f76460 0x560148f88ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4939448700 1 -- 172.21.15.115:0/371879590 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560148d1a200 con 0x560148f75860 2023-07-14T18:11:57.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4939c49700 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 0x560148f6f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:57.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f493a44a700 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f76460 0x560148f88ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:57.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4939448700 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f927b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4928005cb0 tx=0x7f4928004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e161e6a13f54c6dc server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4930ff9700 1 -- 172.21.15.115:0/371879590 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928095ea0 con 0x560148f75860 2023-07-14T18:11:57.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4930ff9700 1 -- 172.21.15.115:0/371879590 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f492809f2c0 con 0x560148f75860 2023-07-14T18:11:57.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4930ff9700 1 -- 172.21.15.115:0/371879590 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492809a780 con 0x560148f75860 2023-07-14T18:11:57.386 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4948045b80 1 -- 172.21.15.115:0/371879590 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 msgr2=0x560148f927b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.342+0000 7f4948045b80 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f927b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4928005cb0 tx=0x7f4928004320 comp rx=0 tx=0).stop 2023-07-14T18:11:57.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 -- 172.21.15.115:0/371879590 shutdown_connections 2023-07-14T18:11:57.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f76460 0x560148f88ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f74f10 0x560148f6f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 --2- 172.21.15.115:0/371879590 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f927b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 -- 172.21.15.115:0/371879590 >> 172.21.15.115:0/371879590 conn(0x560148f6a630 msgr2=0x560148f6bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.387 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 -- 172.21.15.115:0/371879590 shutdown_connections 2023-07-14T18:11:57.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 -- 172.21.15.115:0/371879590 wait complete. 2023-07-14T18:11:57.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 -- start start 2023-07-14T18:11:57.388 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 Processor -- start 2023-07-14T18:11:57.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 0x560148f8dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f8e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f76460 0x560148f91040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.391 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148f91580 con 0x560148f75860 2023-07-14T18:11:57.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148f916f0 con 0x560148f74f10 2023-07-14T18:11:57.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x560148f918b0 con 0x560148f76460 2023-07-14T18:11:57.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4948045b80 10 client.0 _ll_register_callbacks cb 0x7f492400b960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:11:57.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4939448700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f8e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4939448700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f8e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54266/0 (socket says 172.21.15.115:54266) 2023-07-14T18:11:57.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.343+0000 7f4939448700 1 -- 172.21.15.115:0/1607897626 learned_addr learned my addr 172.21.15.115:0/1607897626 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:57.392 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f493a44a700 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f76460 0x560148f91040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4939c49700 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 0x560148f8dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4939448700 1 -- 172.21.15.115:0/1607897626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f76460 msgr2=0x560148f91040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4939448700 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f76460 0x560148f91040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4939448700 1 -- 172.21.15.115:0/1607897626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 msgr2=0x560148f8dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:11:57.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4939448700 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 0x560148f8dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4939448700 1 -- 172.21.15.115:0/1607897626 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493419ffa0 con 0x560148f75860 2023-07-14T18:11:57.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4939448700 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f8e020 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4928000c00 tx=0x7f4928005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226775a4e31d1df server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:57.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f49197fa700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2023-07-14T18:11:57.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f49197fa700 1 -- 172.21.15.115:0/1607897626 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4928095ea0 con 0x560148f75860 2023-07-14T18:11:57.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f49197fa700 1 -- 172.21.15.115:0/1607897626 <== mon.0 v2:172.21.15.115:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f49280b9500 con 0x560148f75860 2023-07-14T18:11:57.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4948045b80 1 -- 172.21.15.115:0/1607897626 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560148e1f4a0 con 0x560148f75860 2023-07-14T18:11:57.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f49197fa700 1 -- 172.21.15.115:0/1607897626 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49280a1170 con 0x560148f75860 2023-07-14T18:11:57.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4948045b80 10 client.15435 fetch_fsmap learned FSMap version 24 2023-07-14T18:11:57.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f4948045b80 1 -- 172.21.15.115:0/1607897626 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560148cf2260 con 0x560148f75860 2023-07-14T18:11:57.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f49197fa700 1 -- 172.21.15.115:0/1607897626 <== mon.0 v2:172.21.15.115:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x560148e1f4a0 con 0x560148f75860 2023-07-14T18:11:57.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.344+0000 7f49197fa700 1 -- 172.21.15.115:0/1607897626 <== mon.0 v2:172.21.15.115:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4928084ae0 con 0x560148f75860 2023-07-14T18:11:57.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f4948045b80 10 client.15435 fetch_fsmap finished waiting for FSMap version 24 2023-07-14T18:11:57.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f4948045b80 10 client.15435 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:11:57.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f4948045b80 20 client.15435 populate_metadata read hostname 'smithi115' 2023-07-14T18:11:57.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f4948045b80 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x560148f89a60 0x560148fdf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:57.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f4948045b80 4 client.15435 mds_command: new command op to 24515 tid=0[{"prefix": "status"}] 2023-07-14T18:11:57.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f4948045b80 1 -- 172.21.15.115:0/1607897626 --> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560148e81000 con 0x560148f89a60 2023-07-14T18:11:57.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f4939c49700 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x560148f89a60 0x560148fdf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:57.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f49197fa700 10 client.15435 ms_handle_connect on v2:172.21.15.150:6838/2895627668 2023-07-14T18:11:57.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f4939c49700 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x560148f89a60 0x560148fdf6d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e839295660ca9bb server_cookie=fd0b925e9142c229 in_seq=0 out_seq=0 2023-07-14T18:11:57.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f49197fa700 1 -- 172.21.15.115:0/1607897626 <== mds.0 v2:172.21.15.150:6838/2895627668 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x560148e81000 con 0x560148f89a60 2023-07-14T18:11:57.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.345+0000 7f49197fa700 10 client.15435 handle_command_reply: tid=0 2023-07-14T18:11:57.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 client.15435 shutdown 2023-07-14T18:11:57.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 -- 172.21.15.115:0/1607897626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 msgr2=0x560148f8e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f8e020 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4928000c00 tx=0x7f4928005ad0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 -- 172.21.15.115:0/1607897626 shutdown_connections 2023-07-14T18:11:57.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.150:6838/2895627668,v1:172.21.15.150:6839/2895627668] conn(0x560148f89a60 0x560148fdf6d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x560148f76460 0x560148f91040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560148f74f10 0x560148f8dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.397 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 --2- 172.21.15.115:0/1607897626 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x560148f75860 0x560148f8e020 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f49197fa700 0 client.15435 ms_handle_reset on v2:172.21.15.150:6838/2895627668 2023-07-14T18:11:57.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 -- 172.21.15.115:0/1607897626 >> 172.21.15.115:0/1607897626 conn(0x560148f6a630 msgr2=0x560148f6f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 -- 172.21.15.115:0/1607897626 shutdown_connections 2023-07-14T18:11:57.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 1 -- 172.21.15.115:0/1607897626 wait complete. 2023-07-14T18:11:57.398 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f4948045b80 20 client.15435 trim_cache size 0 max 16384 2023-07-14T18:11:57.398 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:11:57.399 INFO:teuthology.orchestra.run.smithi115.stdout: "cluster_fsid": "a39f385c-2270-11ee-9b33-001a4aab830c", 2023-07-14T18:11:57.399 INFO:teuthology.orchestra.run.smithi115.stdout: "whoami": 0, 2023-07-14T18:11:57.399 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 24515, 2023-07-14T18:11:57.399 INFO:teuthology.orchestra.run.smithi115.stdout: "want_state": "up:active", 2023-07-14T18:11:57.399 INFO:teuthology.orchestra.run.smithi115.stdout: "state": "up:active", 2023-07-14T18:11:57.400 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs", 2023-07-14T18:11:57.400 INFO:teuthology.orchestra.run.smithi115.stdout: "rank_uptime": 98.935760637000001, 2023-07-14T18:11:57.400 INFO:teuthology.orchestra.run.smithi115.stdout: "mdsmap_epoch": 24, 2023-07-14T18:11:57.400 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch": 89, 2023-07-14T18:11:57.400 INFO:teuthology.orchestra.run.smithi115.stdout: "osdmap_epoch_barrier": 87, 2023-07-14T18:11:57.400 INFO:teuthology.orchestra.run.smithi115.stdout: "uptime": 105.787720373 2023-07-14T18:11:57.400 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:11:57.401 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 -- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f491c0a1260 msgr2=0x7f491c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f491c0a1260 0x7f491c0a3710 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f492c009d10 tx=0x7f492c009040 comp rx=0 tx=0).stop 2023-07-14T18:11:57.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 -- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 msgr2=0x7f49340fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:57.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 0x7f49340fe640 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4928006010 tx=0x7f4928004730 comp rx=0 tx=0).stop 2023-07-14T18:11:57.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 -- 172.21.15.115:0/2047333087 shutdown_connections 2023-07-14T18:11:57.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f491c0a1260 0x7f491c0a3710 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.402 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49341058d0 0x7f49340feb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4934100690 0x7f4934100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 --2- 172.21.15.115:0/2047333087 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4934102fb0 0x7f49340fe640 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:57.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 -- 172.21.15.115:0/2047333087 >> 172.21.15.115:0/2047333087 conn(0x7f49340fa130 msgr2=0x7f49341012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:57.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 -- 172.21.15.115:0/2047333087 shutdown_connections 2023-07-14T18:11:57.403 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:57.346+0000 7f493bead700 1 -- 172.21.15.115:0/2047333087 wait complete. 2023-07-14T18:11:57.403 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-07-14T18:11:57.414 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-07-14T18:11:57.415 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2023-07-14T18:11:57.415 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2023-07-14T18:11:57.415 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T18:11:57.416 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T18:11:57.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:11:57.416 DEBUG:teuthology.orchestra.run.smithi115:> ip netns list 2023-07-14T18:11:57.433 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-07-14T18:11:57.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:11:57.434 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-14T18:11:57.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:11:57.513 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link delete ceph-brx 2023-07-14T18:11:57.610 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-07-14T18:11:57.611 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid a39f385c-2270-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T18:11:57.857 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:57 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3169433311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:57.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:57 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3467032915' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:57.858 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:57 smithi115 ceph-mon[127451]: pgmap v303: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T18:11:57.872 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:57 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3169433311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:57.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:57 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3467032915' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:57.873 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:57 smithi150 ceph-mon[128853]: pgmap v303: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T18:11:57.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:57 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3169433311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:11:57.935 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:57 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3467032915' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:11:57.936 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:57 smithi187 ceph-mon[129317]: pgmap v303: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T18:11:58.162 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:11:59.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.429+0000 7f4899e79700 1 -- 172.21.15.115:0/1739204915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 msgr2=0x7f4894102f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:59.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.429+0000 7f4899e79700 1 --2- 172.21.15.115:0/1739204915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 0x7f4894102f50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f48880099c0 tx=0x7f4888009cd0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.429+0000 7f4899e79700 1 -- 172.21.15.115:0/1739204915 shutdown_connections 2023-07-14T18:11:59.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.429+0000 7f4899e79700 1 --2- 172.21.15.115:0/1739204915 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4894103680 0x7f489410b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.429+0000 7f4899e79700 1 --2- 172.21.15.115:0/1739204915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48940fe5e0 0x7f48940fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.429+0000 7f4899e79700 1 --2- 172.21.15.115:0/1739204915 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 0x7f4894102f50 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.429+0000 7f4899e79700 1 -- 172.21.15.115:0/1739204915 >> 172.21.15.115:0/1739204915 conn(0x7f4894075280 msgr2=0x7f4894075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:59.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.429+0000 7f4899e79700 1 -- 172.21.15.115:0/1739204915 shutdown_connections 2023-07-14T18:11:59.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.429+0000 7f4899e79700 1 -- 172.21.15.115:0/1739204915 wait complete. 2023-07-14T18:11:59.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4899e79700 1 Processor -- start 2023-07-14T18:11:59.472 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4899e79700 1 -- start start 2023-07-14T18:11:59.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4899e79700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48940fe5e0 0x7f48940ffff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:59.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4899e79700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 0x7f4894100530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:59.473 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4899e79700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4894103680 0x7f4894100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:59.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4899e79700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48941a4120 con 0x7f48940fef80 2023-07-14T18:11:59.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4899e79700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48941a4290 con 0x7f4894103680 2023-07-14T18:11:59.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4899e79700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48941a4450 con 0x7f48940fe5e0 2023-07-14T18:11:59.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4892ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 0x7f4894100530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:59.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4892ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 0x7f4894100530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54278/0 (socket says 172.21.15.115:54278) 2023-07-14T18:11:59.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4892ffd700 1 -- 172.21.15.115:0/4021212787 learned_addr learned my addr 172.21.15.115:0/4021212787 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:11:59.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4893fff700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4894103680 0x7f4894100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:59.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f48937fe700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48940fe5e0 0x7f48940ffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:59.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4892ffd700 1 -- 172.21.15.115:0/4021212787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48940fe5e0 msgr2=0x7f48940ffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:59.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.430+0000 7f4892ffd700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48940fe5e0 0x7f48940ffff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f4892ffd700 1 -- 172.21.15.115:0/4021212787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4894103680 msgr2=0x7f4894100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:59.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f4892ffd700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4894103680 0x7f4894100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f4892ffd700 1 -- 172.21.15.115:0/4021212787 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48880095f0 con 0x7f48940fef80 2023-07-14T18:11:59.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f48937fe700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48940fe5e0 0x7f48940ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:11:59.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f4892ffd700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 0x7f4894100530 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4888006010 tx=0x7f4888004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:59.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f4890ff9700 1 -- 172.21.15.115:0/4021212787 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488801d070 con 0x7f48940fef80 2023-07-14T18:11:59.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f4890ff9700 1 -- 172.21.15.115:0/4021212787 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4888004c70 con 0x7f48940fef80 2023-07-14T18:11:59.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48941a46d0 con 0x7f48940fef80 2023-07-14T18:11:59.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f4890ff9700 1 -- 172.21.15.115:0/4021212787 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888005290 con 0x7f48940fef80 2023-07-14T18:11:59.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.431+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48941a4ae0 con 0x7f48940fef80 2023-07-14T18:11:59.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.432+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4894107960 con 0x7f48940fef80 2023-07-14T18:11:59.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.433+0000 7f4890ff9700 1 -- 172.21.15.115:0/4021212787 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f4888003c00 con 0x7f48940fef80 2023-07-14T18:11:59.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.433+0000 7f4890ff9700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f487c0a11e0 0x7f487c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:11:59.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.433+0000 7f4890ff9700 1 -- 172.21.15.115:0/4021212787 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f48880cdfd0 con 0x7f48940fef80 2023-07-14T18:11:59.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.434+0000 7f48937fe700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f487c0a11e0 0x7f487c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:11:59.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.434+0000 7f48937fe700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f487c0a11e0 0x7f487c0a3690 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4884005fd0 tx=0x7f4884005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:11:59.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.437+0000 7f4890ff9700 1 -- 172.21.15.115:0/4021212787 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4888097340 con 0x7f48940fef80 2023-07-14T18:11:59.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.665+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f489404e340 con 0x7f48940fef80 2023-07-14T18:11:59.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.666+0000 7f4890ff9700 1 -- 172.21.15.115:0/4021212787 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+88 (secure 0 0 0) 0x7f488809a960 con 0x7f48940fef80 2023-07-14T18:11:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-14T18:11:59.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f487c0a11e0 msgr2=0x7f487c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:59.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f487c0a11e0 0x7f487c0a3690 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4884005fd0 tx=0x7f4884005ee0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 msgr2=0x7f4894100530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:11:59.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 0x7f4894100530 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4888006010 tx=0x7f4888004730 comp rx=0 tx=0).stop 2023-07-14T18:11:59.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 shutdown_connections 2023-07-14T18:11:59.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f487c0a11e0 0x7f487c0a3690 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4894103680 0x7f4894100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48940fe5e0 0x7f48940ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 --2- 172.21.15.115:0/4021212787 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f48940fef80 0x7f4894100530 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:11:59.709 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 >> 172.21.15.115:0/4021212787 conn(0x7f4894075280 msgr2=0x7f489410b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:11:59.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 shutdown_connections 2023-07-14T18:11:59.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:11:59.668+0000 7f4899e79700 1 -- 172.21.15.115:0/4021212787 wait complete. 2023-07-14T18:12:00.208 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-14T18:12:00.208 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T18:12:00.208 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi115.front.sepia.ceph.com 2023-07-14T18:12:00.209 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-14T18:12:00.209 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T18:12:00.209 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-14T18:12:00.209 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T18:12:00.209 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T18:12:00.210 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-14T18:12:00.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.210 DEBUG:teuthology.orchestra.run.smithi115:> ip addr 2023-07-14T18:12:00.228 INFO:teuthology.orchestra.run.smithi115.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T18:12:00.229 INFO:teuthology.orchestra.run.smithi115.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T18:12:00.229 INFO:teuthology.orchestra.run.smithi115.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T18:12:00.229 INFO:teuthology.orchestra.run.smithi115.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:12:00.229 INFO:teuthology.orchestra.run.smithi115.stdout: inet6 ::1/128 scope host 2023-07-14T18:12:00.229 INFO:teuthology.orchestra.run.smithi115.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:12:00.230 INFO:teuthology.orchestra.run.smithi115.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:12:00.230 INFO:teuthology.orchestra.run.smithi115.stdout: link/ether 0c:c4:7a:d8:a5:44 brd ff:ff:ff:ff:ff:ff 2023-07-14T18:12:00.230 INFO:teuthology.orchestra.run.smithi115.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:12:00.230 INFO:teuthology.orchestra.run.smithi115.stdout: link/ether 0c:c4:7a:d8:a5:45 brd ff:ff:ff:ff:ff:ff 2023-07-14T18:12:00.230 INFO:teuthology.orchestra.run.smithi115.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:12:00.230 INFO:teuthology.orchestra.run.smithi115.stdout: link/ether 0c:c4:7a:8f:ce:8e brd ff:ff:ff:ff:ff:ff 2023-07-14T18:12:00.230 INFO:teuthology.orchestra.run.smithi115.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T18:12:00.230 INFO:teuthology.orchestra.run.smithi115.stdout: link/ether 0c:c4:7a:8f:ce:8f brd ff:ff:ff:ff:ff:ff 2023-07-14T18:12:00.231 INFO:teuthology.orchestra.run.smithi115.stdout: inet 172.21.15.115/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-14T18:12:00.231 INFO:teuthology.orchestra.run.smithi115.stdout: valid_lft 41390sec preferred_lft 41390sec 2023-07-14T18:12:00.231 INFO:teuthology.orchestra.run.smithi115.stdout: inet6 fe80::1d33:d16a:ed86:b5bf/64 scope link noprefixroute 2023-07-14T18:12:00.231 INFO:teuthology.orchestra.run.smithi115.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:12:00.231 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T18:12:00.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.232 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:12:00.232 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:12:00.232 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link add name ceph-brx type bridge 2023-07-14T18:12:00.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip addr flush dev ceph-brx 2023-07-14T18:12:00.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link set ceph-brx up 2023-07-14T18:12:00.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T18:12:00.233 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:12:00.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:59 smithi115 ceph-mon[127451]: pgmap v304: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-07-14T18:12:00.309 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:11:59 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4021212787' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:12:00.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.350 DEBUG:teuthology.orchestra.run.smithi115:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T18:12:00.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:59 smithi150 ceph-mon[128853]: pgmap v304: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-07-14T18:12:00.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:11:59 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/4021212787' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:12:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:1 2023-07-14T18:12:00.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.420 DEBUG:teuthology.orchestra.run.smithi115:> route 2023-07-14T18:12:00.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:59 smithi187 ceph-mon[129317]: pgmap v304: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-07-14T18:12:00.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:11:59 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/4021212787' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:12:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:Kernel IP routing table 2023-07-14T18:12:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T18:12:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-14T18:12:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-14T18:12:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T18:12:00.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.480 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:12:00.480 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:12:00.480 DEBUG:teuthology.orchestra.run.smithi115:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-14T18:12:00.480 DEBUG:teuthology.orchestra.run.smithi115:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-14T18:12:00.480 DEBUG:teuthology.orchestra.run.smithi115:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-14T18:12:00.481 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:12:00.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.585 DEBUG:teuthology.orchestra.run.smithi115:> ip netns list 2023-07-14T18:12:00.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.603 DEBUG:teuthology.orchestra.run.smithi115:> ip netns list-id 2023-07-14T18:12:00.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.660 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:12:00.660 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:12:00.660 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T18:12:00.660 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-07-14T18:12:00.661 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:12:00.754 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-07-14T18:12:00.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.754 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:12:00.755 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:12:00.755 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-07-14T18:12:00.755 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T18:12:00.755 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-07-14T18:12:00.755 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-07-14T18:12:00.756 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-07-14T18:12:00.756 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:12:00.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:00.949 DEBUG:teuthology.orchestra.run.smithi115:> (cd / && exec bash -c ' 2023-07-14T18:12:00.949 DEBUG:teuthology.orchestra.run.smithi115:> set -e 2023-07-14T18:12:00.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link set brx.0 up 2023-07-14T18:12:00.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T18:12:00.950 DEBUG:teuthology.orchestra.run.smithi115:> ') 2023-07-14T18:12:01.058 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2023-07-14T18:12:01.058 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T18:12:01.059 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-14T18:12:01.082 INFO:teuthology.orchestra.run.smithi115.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-14T18:12:01.082 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T18:12:01.083 DEBUG:teuthology.orchestra.run.smithi115:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-14T18:12:01.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:00 smithi115 ceph-mon[127451]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T18:12:01.123 DEBUG:teuthology.orchestra.run.smithi115:> sudo modprobe fuse 2023-07-14T18:12:01.191 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T18:12:01.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T18:12:01.258 INFO:teuthology.orchestra.run.smithi115.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T18:12:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T18:12:01.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:01.260 DEBUG:teuthology.orchestra.run.smithi115:> ls /sys/fs/fuse/connections 2023-07-14T18:12:01.318 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-14T18:12:01.318 DEBUG:teuthology.orchestra.run.smithi115:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/sv_0/0ce1c3df-0ddd-4436-9361-8f440b239633 --client_fs=cephfs) 2023-07-14T18:12:01.361 DEBUG:teuthology.orchestra.run.smithi115:> sudo modprobe fuse 2023-07-14T18:12:01.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:00 smithi150 ceph-mon[128853]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T18:12:01.387 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T18:12:01.388 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T18:12:01.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:00 smithi187 ceph-mon[129317]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T18:12:01.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi115.stderr:2023-07-14T18:12:01.408+0000 7f0e94a725c0 -1 init, newargv = 0x558166ef52c0 newargc=15 2023-07-14T18:12:01.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi115.stderr:ceph-fuse[177428]: starting ceph client 2023-07-14T18:12:01.455 INFO:teuthology.orchestra.run.smithi115.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T18:12:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T18:12:01.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:01.457 DEBUG:teuthology.orchestra.run.smithi115:> ls /sys/fs/fuse/connections 2023-07-14T18:12:01.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi115.stderr:ceph-fuse[177428]: starting fuse 2023-07-14T18:12:01.481 INFO:teuthology.orchestra.run.smithi115.stdout:124 2023-07-14T18:12:01.481 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-07-14T18:12:01.482 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-14T18:12:01.482 DEBUG:teuthology.orchestra.run.smithi115:> import glob 2023-07-14T18:12:01.482 DEBUG:teuthology.orchestra.run.smithi115:> import re 2023-07-14T18:12:01.482 DEBUG:teuthology.orchestra.run.smithi115:> import os 2023-07-14T18:12:01.483 DEBUG:teuthology.orchestra.run.smithi115:> import subprocess 2023-07-14T18:12:01.483 DEBUG:teuthology.orchestra.run.smithi115:> 2023-07-14T18:12:01.483 DEBUG:teuthology.orchestra.run.smithi115:> def _find_admin_socket(client_name): 2023-07-14T18:12:01.483 DEBUG:teuthology.orchestra.run.smithi115:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-07-14T18:12:01.484 DEBUG:teuthology.orchestra.run.smithi115:> files = glob.glob(asok_path) 2023-07-14T18:12:01.484 DEBUG:teuthology.orchestra.run.smithi115:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-07-14T18:12:01.484 DEBUG:teuthology.orchestra.run.smithi115:> 2023-07-14T18:12:01.484 DEBUG:teuthology.orchestra.run.smithi115:> # Given a non-glob path, it better be there 2023-07-14T18:12:01.485 DEBUG:teuthology.orchestra.run.smithi115:> if "*" not in asok_path: 2023-07-14T18:12:01.485 DEBUG:teuthology.orchestra.run.smithi115:> assert(len(files) == 1) 2023-07-14T18:12:01.485 DEBUG:teuthology.orchestra.run.smithi115:> return files[0] 2023-07-14T18:12:01.485 DEBUG:teuthology.orchestra.run.smithi115:> 2023-07-14T18:12:01.485 DEBUG:teuthology.orchestra.run.smithi115:> for f in files: 2023-07-14T18:12:01.486 DEBUG:teuthology.orchestra.run.smithi115:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-14T18:12:01.486 DEBUG:teuthology.orchestra.run.smithi115:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-14T18:12:01.486 DEBUG:teuthology.orchestra.run.smithi115:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-14T18:12:01.486 DEBUG:teuthology.orchestra.run.smithi115:> contents = proc_f.read() 2023-07-14T18:12:01.487 DEBUG:teuthology.orchestra.run.smithi115:> if mountpoint in contents: 2023-07-14T18:12:01.487 DEBUG:teuthology.orchestra.run.smithi115:> return f 2023-07-14T18:12:01.487 DEBUG:teuthology.orchestra.run.smithi115:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-14T18:12:01.487 DEBUG:teuthology.orchestra.run.smithi115:> 2023-07-14T18:12:01.487 DEBUG:teuthology.orchestra.run.smithi115:> print(_find_admin_socket("client.0")) 2023-07-14T18:12:01.488 DEBUG:teuthology.orchestra.run.smithi115:> ' 2023-07-14T18:12:01.605 INFO:teuthology.orchestra.run.smithi115.stdout:/var/run/ceph/ceph-client.0.177428.asok 2023-07-14T18:12:01.787 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.177428.asok 2023-07-14T18:12:01.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:01.788 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.177428.asok status 2023-07-14T18:12:01.944 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2023-07-14T18:12:01.944 INFO:teuthology.orchestra.run.smithi115.stdout: "metadata": { 2023-07-14T18:12:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_sha1": "81c9387d2e7576da065747ec9c16d1c7d69016ec", 2023-07-14T18:12:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:12:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "entity_id": "0", 2023-07-14T18:12:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "hostname": "smithi115", 2023-07-14T18:12:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-07-14T18:12:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "pid": "177428", 2023-07-14T18:12:01.945 INFO:teuthology.orchestra.run.smithi115.stdout: "root": "/volumes/_nogroup/sv_0/0ce1c3df-0ddd-4436-9361-8f440b239633" 2023-07-14T18:12:01.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-07-14T18:12:01.946 INFO:teuthology.orchestra.run.smithi115.stdout: "dentry_count": 0, 2023-07-14T18:12:01.946 INFO:teuthology.orchestra.run.smithi115.stdout: "dentry_pinned_count": 0, 2023-07-14T18:12:01.946 INFO:teuthology.orchestra.run.smithi115.stdout: "id": 15447, 2023-07-14T18:12:01.946 INFO:teuthology.orchestra.run.smithi115.stdout: "inst": { 2023-07-14T18:12:01.946 INFO:teuthology.orchestra.run.smithi115.stdout: "name": { 2023-07-14T18:12:01.946 INFO:teuthology.orchestra.run.smithi115.stdout: "type": "client", 2023-07-14T18:12:01.946 INFO:teuthology.orchestra.run.smithi115.stdout: "num": 15447 2023-07-14T18:12:01.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-07-14T18:12:01.947 INFO:teuthology.orchestra.run.smithi115.stdout: "addr": { 2023-07-14T18:12:01.947 INFO:teuthology.orchestra.run.smithi115.stdout: "type": "v1", 2023-07-14T18:12:01.947 INFO:teuthology.orchestra.run.smithi115.stdout: "addr": "192.168.0.1:0", 2023-07-14T18:12:01.947 INFO:teuthology.orchestra.run.smithi115.stdout: "nonce": 2973515422 2023-07-14T18:12:01.947 INFO:teuthology.orchestra.run.smithi115.stdout: } 2023-07-14T18:12:01.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-07-14T18:12:01.947 INFO:teuthology.orchestra.run.smithi115.stdout: "addr": { 2023-07-14T18:12:01.948 INFO:teuthology.orchestra.run.smithi115.stdout: "type": "v1", 2023-07-14T18:12:01.948 INFO:teuthology.orchestra.run.smithi115.stdout: "addr": "192.168.0.1:0", 2023-07-14T18:12:01.948 INFO:teuthology.orchestra.run.smithi115.stdout: "nonce": 2973515422 2023-07-14T18:12:01.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-07-14T18:12:01.948 INFO:teuthology.orchestra.run.smithi115.stdout: "inst_str": "client.15447 192.168.0.1:0/2973515422", 2023-07-14T18:12:01.948 INFO:teuthology.orchestra.run.smithi115.stdout: "addr_str": "192.168.0.1:0/2973515422", 2023-07-14T18:12:01.948 INFO:teuthology.orchestra.run.smithi115.stdout: "inode_count": 5, 2023-07-14T18:12:01.948 INFO:teuthology.orchestra.run.smithi115.stdout: "mds_epoch": 24, 2023-07-14T18:12:01.949 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_epoch": 89, 2023-07-14T18:12:01.949 INFO:teuthology.orchestra.run.smithi115.stdout: "osd_epoch_barrier": 0, 2023-07-14T18:12:01.949 INFO:teuthology.orchestra.run.smithi115.stdout: "blocklisted": false, 2023-07-14T18:12:01.949 INFO:teuthology.orchestra.run.smithi115.stdout: "fs_name": "cephfs" 2023-07-14T18:12:01.949 INFO:teuthology.orchestra.run.smithi115.stdout:} 2023-07-14T18:12:01.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:01.955 DEBUG:teuthology.orchestra.run.smithi115:> stat --file-system '--printf=%T 2023-07-14T18:12:01.955 DEBUG:teuthology.orchestra.run.smithi115:> ' -- /home/ubuntu/cephtest/mnt.0 2023-07-14T18:12:01.972 INFO:teuthology.orchestra.run.smithi115.stdout:fuseblk 2023-07-14T18:12:01.972 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-07-14T18:12:01.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:01.973 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-14T18:12:02.041 INFO:teuthology.run_tasks:Running task exec... 2023-07-14T18:12:02.054 INFO:teuthology.task.exec:Executing custom commands... 2023-07-14T18:12:02.055 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi115.front.sepia.ceph.com 2023-07-14T18:12:02.055 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2023-07-14T18:12:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.166+0000 7fe05896f700 1 -- 172.21.15.115:0/3464716691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0541006f0 msgr2=0x7fe054106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.166+0000 7fe05896f700 1 --2- 172.21.15.115:0/3464716691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0541006f0 0x7fe054106fb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00b0a0 tx=0x7fe04c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.166+0000 7fe05896f700 1 -- 172.21.15.115:0/3464716691 shutdown_connections 2023-07-14T18:12:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.166+0000 7fe05896f700 1 --2- 172.21.15.115:0/3464716691 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0541006f0 0x7fe054106fb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.166+0000 7fe05896f700 1 --2- 172.21.15.115:0/3464716691 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0540feb70 0x7fe0540fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:02.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.166+0000 7fe05896f700 1 --2- 172.21.15.115:0/3464716691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0540ffd60 0x7fe0541001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:02.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.166+0000 7fe05896f700 1 -- 172.21.15.115:0/3464716691 >> 172.21.15.115:0/3464716691 conn(0x7fe0540fa190 msgr2=0x7fe0540fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:02.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.166+0000 7fe05896f700 1 -- 172.21.15.115:0/3464716691 shutdown_connections 2023-07-14T18:12:02.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.166+0000 7fe05896f700 1 -- 172.21.15.115:0/3464716691 wait complete. 2023-07-14T18:12:02.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05896f700 1 Processor -- start 2023-07-14T18:12:02.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05896f700 1 -- start start 2023-07-14T18:12:02.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05896f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0540feb70 0x7fe054194ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:02.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05896f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0540ffd60 0x7fe054194ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:02.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05896f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0541006f0 0x7fe054195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:02.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05896f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0541964d0 con 0x7fe0540feb70 2023-07-14T18:12:02.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05896f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe054196640 con 0x7fe0540ffd60 2023-07-14T18:12:02.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05896f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe054196800 con 0x7fe0541006f0 2023-07-14T18:12:02.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0540feb70 0x7fe054194ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:02.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0540feb70 0x7fe054194ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54308/0 (socket says 172.21.15.115:54308) 2023-07-14T18:12:02.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe05259c700 1 -- 172.21.15.115:0/3327683012 learned_addr learned my addr 172.21.15.115:0/3327683012 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:02.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.167+0000 7fe051d9b700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0540ffd60 0x7fe054194ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:02.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe05259c700 1 -- 172.21.15.115:0/3327683012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0541006f0 msgr2=0x7fe054195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:02.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe05259c700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0541006f0 0x7fe054195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:02.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe05259c700 1 -- 172.21.15.115:0/3327683012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0540ffd60 msgr2=0x7fe054194ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:02.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe05259c700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0540ffd60 0x7fe054194ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:02.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe05259c700 1 -- 172.21.15.115:0/3327683012 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c009d00 con 0x7fe0540feb70 2023-07-14T18:12:02.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe05259c700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0540feb70 0x7fe054194ab0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe04400b500 tx=0x7fe04400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:02.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe0437fe700 1 -- 172.21.15.115:0/3327683012 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe044011830 con 0x7fe0540feb70 2023-07-14T18:12:02.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe0437fe700 1 -- 172.21.15.115:0/3327683012 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe044011e20 con 0x7fe0540feb70 2023-07-14T18:12:02.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe054196ae0 con 0x7fe0540feb70 2023-07-14T18:12:02.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe0437fe700 1 -- 172.21.15.115:0/3327683012 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe044010dd0 con 0x7fe0540feb70 2023-07-14T18:12:02.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.168+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05419d320 con 0x7fe0540feb70 2023-07-14T18:12:02.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.169+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05404e420 con 0x7fe0540feb70 2023-07-14T18:12:02.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.170+0000 7fe0437fe700 1 -- 172.21.15.115:0/3327683012 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fe044011990 con 0x7fe0540feb70 2023-07-14T18:12:02.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.170+0000 7fe0437fe700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe03c0a1150 0x7fe03c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:02.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.170+0000 7fe051d9b700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe03c0a1150 0x7fe03c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:02.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.171+0000 7fe0437fe700 1 -- 172.21.15.115:0/3327683012 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe0440cc6f0 con 0x7fe0540feb70 2023-07-14T18:12:02.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.171+0000 7fe051d9b700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe03c0a1150 0x7fe03c0a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe048005950 tx=0x7fe0480058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:02.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.175+0000 7fe0437fe700 1 -- 172.21.15.115:0/3327683012 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe044095b30 con 0x7fe0540feb70 2023-07-14T18:12:02.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:01 smithi115 ceph-mon[127451]: pgmap v305: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-14T18:12:02.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:01 smithi150 ceph-mon[128853]: pgmap v305: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-14T18:12:02.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:01 smithi187 ceph-mon[129317]: pgmap v305: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-14T18:12:02.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.411+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fe054105a40 con 0x7fe0540feb70 2023-07-14T18:12:03.026 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.986+0000 7fe0437fe700 1 -- 172.21.15.115:0/3327683012 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe044099150 con 0x7fe0540feb70 2023-07-14T18:12:03.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe03c0a1150 msgr2=0x7fe03c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:03.028 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe03c0a1150 0x7fe03c0a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe048005950 tx=0x7fe0480058e0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0540feb70 msgr2=0x7fe054194ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:03.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0540feb70 0x7fe054194ab0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe04400b500 tx=0x7fe04400b8c0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 shutdown_connections 2023-07-14T18:12:03.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe03c0a1150 0x7fe03c0a3600 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0541006f0 0x7fe054195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe0540feb70 0x7fe054194ab0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.029 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 --2- 172.21.15.115:0/3327683012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0540ffd60 0x7fe054194ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 >> 172.21.15.115:0/3327683012 conn(0x7fe0540fa190 msgr2=0x7fe0540fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:03.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 shutdown_connections 2023-07-14T18:12:03.030 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:02.988+0000 7fe05896f700 1 -- 172.21.15.115:0/3327683012 wait complete. 2023-07-14T18:12:03.042 DEBUG:teuthology.orchestra.run.smithi115:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01' 2023-07-14T18:12:03.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 -- 172.21.15.115:0/3155866558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bf40feb30 msgr2=0x7f5bf40fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:03.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 --2- 172.21.15.115:0/3155866558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bf40feb30 0x7f5bf40fef40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009990 tx=0x7f5be4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 -- 172.21.15.115:0/3155866558 shutdown_connections 2023-07-14T18:12:03.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 --2- 172.21.15.115:0/3155866558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf4100770 0x7f5bf4107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 --2- 172.21.15.115:0/3155866558 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bf40feb30 0x7f5bf40fef40 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 --2- 172.21.15.115:0/3155866558 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf40ffdc0 0x7f5bf4100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 -- 172.21.15.115:0/3155866558 >> 172.21.15.115:0/3155866558 conn(0x7f5bf40fa130 msgr2=0x7f5bf40fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:03.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 -- 172.21.15.115:0/3155866558 shutdown_connections 2023-07-14T18:12:03.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 -- 172.21.15.115:0/3155866558 wait complete. 2023-07-14T18:12:03.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 Processor -- start 2023-07-14T18:12:03.171 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.129+0000 7f5bfc192700 1 -- start start 2023-07-14T18:12:03.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bfc192700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bf40feb30 0x7f5bf41948f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:03.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bfc192700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf40ffdc0 0x7f5bf4194e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:03.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bfc192700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf4100770 0x7f5bf4195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:03.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bfc192700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4196310 con 0x7f5bf40feb30 2023-07-14T18:12:03.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bfc192700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4196480 con 0x7f5bf40ffdc0 2023-07-14T18:12:03.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bfc192700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4196640 con 0x7f5bf4100770 2023-07-14T18:12:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bf9f2e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bf40feb30 0x7f5bf41948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bf9f2e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bf40feb30 0x7f5bf41948f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54322/0 (socket says 172.21.15.115:54322) 2023-07-14T18:12:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bf9f2e700 1 -- 172.21.15.115:0/2977736684 learned_addr learned my addr 172.21.15.115:0/2977736684 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bfa72f700 1 --2- 172.21.15.115:0/2977736684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf4100770 0x7f5bf4195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bf9f2e700 1 -- 172.21.15.115:0/2977736684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf4100770 msgr2=0x7f5bf4195cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bf9f2e700 1 --2- 172.21.15.115:0/2977736684 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5bf4100770 0x7f5bf4195cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bf9f2e700 1 -- 172.21.15.115:0/2977736684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf40ffdc0 msgr2=0x7f5bf4194e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bf9f2e700 1 --2- 172.21.15.115:0/2977736684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf40ffdc0 0x7f5bf4194e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bf9f2e700 1 -- 172.21.15.115:0/2977736684 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be40095f0 con 0x7f5bf40feb30 2023-07-14T18:12:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bf9f2e700 1 --2- 172.21.15.115:0/2977736684 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5bf40feb30 0x7f5bf41948f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5be4006010 tx=0x7f5be4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5beaffd700 1 -- 172.21.15.115:0/2977736684 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be401d070 con 0x7f5bf40feb30 2023-07-14T18:12:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bfc192700 1 -- 172.21.15.115:0/2977736684 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf41968c0 con 0x7f5bf40feb30 2023-07-14T18:12:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5beaffd700 1 -- 172.21.15.115:0/2977736684 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5be40041b0 con 0x7f5bf40feb30 2023-07-14T18:12:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5beaffd700 1 -- 172.21.15.115:0/2977736684 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4022da0 con 0x7f5bf40feb30 2023-07-14T18:12:03.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.130+0000 7f5bfc192700 1 -- 172.21.15.115:0/2977736684 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf419d0e0 con 0x7f5bf40feb30 2023-07-14T18:12:03.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.131+0000 7f5bfc192700 1 -- 172.21.15.115:0/2977736684 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf41061d0 con 0x7f5bf40feb30 2023-07-14T18:12:03.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.132+0000 7f5beaffd700 1 -- 172.21.15.115:0/2977736684 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f5be4022570 con 0x7f5bf40feb30 2023-07-14T18:12:03.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.133+0000 7f5beaffd700 1 --2- 172.21.15.115:0/2977736684 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5be00a1150 0x7f5be00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:03.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.133+0000 7f5bf972d700 1 --2- 172.21.15.115:0/2977736684 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5be00a1150 0x7f5be00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:03.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.133+0000 7f5beaffd700 1 -- 172.21.15.115:0/2977736684 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5be40ce560 con 0x7f5bf40feb30 2023-07-14T18:12:03.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.133+0000 7f5bf972d700 1 --2- 172.21.15.115:0/2977736684 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f5be00a1150 0x7f5be00a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005950 tx=0x7f5bf00058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:03.177 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.136+0000 7f5beaffd700 1 -- 172.21.15.115:0/2977736684 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5be4097a30 con 0x7f5bf40feb30 2023-07-14T18:12:03.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:02 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3327683012' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T18:12:03.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:02 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3327683012' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T18:12:03.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.356+0000 7f5bfc192700 1 -- 172.21.15.115:0/2977736684 --> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bf4060a80 con 0x7f5be00a1150 2023-07-14T18:12:03.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:03.370+0000 7f5beaffd700 1 -- 172.21.15.115:0/2977736684 <== mgr.14150 v2:172.21.15.115:6800/2625350855 1 ==== mgr_command_reply(tid 0: -1 Operation not permitted for group '_nogroup' as it is an internal group.) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5bf4060a80 con 0x7f5be00a1150 2023-07-14T18:12:03.412 INFO:teuthology.orchestra.run.smithi115.stderr:Error EPERM: Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T18:12:03.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:12:03.416 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T18:12:03.480 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7a975a42ae3b42e2afb6beeb8740f302 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi115 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T18:12:03.482 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-07-14T18:12:03.492 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-07-14T18:12:03.493 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:12:03.493 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:12:08.502 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:12:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:12:03.361+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:12:08.503 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:12:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:12:03.361+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:12:08.503 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:12:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:12:03.361+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:12:08.503 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:12:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:12:03.361+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:12:08.503 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:12:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:12:03.361+0000 7f45d8580700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:12:08.503 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:12:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:12:03.369+0000 7f45d8580700 -1 mgr.server reply reply (1) Operation not permitted Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T18:12:08.505 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:03 smithi150 ceph-mon[128853]: pgmap v306: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 8 op/s 2023-07-14T18:12:08.505 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:03 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3327683012' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T18:12:08.505 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:03 smithi150 ceph-mon[128853]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T18:12:08.505 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:04 smithi150 ceph-mon[128853]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:12:08.505 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:05 smithi150 ceph-mon[128853]: pgmap v307: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 852 B/s wr, 6 op/s 2023-07-14T18:12:08.506 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:07 smithi150 ceph-mon[128853]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T18:12:08.506 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:08 smithi150 ceph-mon[128853]: pgmap v308: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 938 B/s wr, 7 op/s 2023-07-14T18:12:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:02 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3327683012' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T18:12:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:03 smithi187 ceph-mon[129317]: pgmap v306: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 8 op/s 2023-07-14T18:12:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:03 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3327683012' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T18:12:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:03 smithi187 ceph-mon[129317]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T18:12:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:04 smithi187 ceph-mon[129317]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:12:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:05 smithi187 ceph-mon[129317]: pgmap v307: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 852 B/s wr, 6 op/s 2023-07-14T18:12:08.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:07 smithi187 ceph-mon[129317]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T18:12:08.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:08 smithi187 ceph-mon[129317]: pgmap v308: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 938 B/s wr, 7 op/s 2023-07-14T18:12:08.508 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:03 smithi115 ceph-mon[127451]: pgmap v306: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 8 op/s 2023-07-14T18:12:08.508 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:03 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3327683012' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T18:12:08.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:03 smithi115 ceph-mon[127451]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T18:12:08.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:04 smithi115 ceph-mon[127451]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:12:08.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:05 smithi115 ceph-mon[127451]: pgmap v307: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 852 B/s wr, 6 op/s 2023-07-14T18:12:08.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:07 smithi115 ceph-mon[127451]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T18:12:08.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:08 smithi115 ceph-mon[127451]: pgmap v308: 129 pgs: 129 active+clean; 640 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 938 B/s wr, 7 op/s 2023-07-14T18:12:08.522 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:12:08.522 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:12:08.583 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:12:08.757 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.716+0000 7fe8a2526700 1 -- 172.21.15.115:0/965688902 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 msgr2=0x7fe89c100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 --2- 172.21.15.115:0/965688902 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 0x7fe89c100170 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe890009990 tx=0x7fe890009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 -- 172.21.15.115:0/965688902 shutdown_connections 2023-07-14T18:12:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 --2- 172.21.15.115:0/965688902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe89c1006b0 0x7fe89c106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.758 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 --2- 172.21.15.115:0/965688902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89c0feb30 0x7fe89c0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 --2- 172.21.15.115:0/965688902 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 0x7fe89c100170 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 -- 172.21.15.115:0/965688902 >> 172.21.15.115:0/965688902 conn(0x7fe89c0fa110 msgr2=0x7fe89c0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 -- 172.21.15.115:0/965688902 shutdown_connections 2023-07-14T18:12:08.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 -- 172.21.15.115:0/965688902 wait complete. 2023-07-14T18:12:08.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 Processor -- start 2023-07-14T18:12:08.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.717+0000 7fe8a2526700 1 -- start start 2023-07-14T18:12:08.761 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8a2526700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89c0feb30 0x7fe89c195790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8a2526700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 0x7fe89c195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8a2526700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe89c1006b0 0x7fe89c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8a2526700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c19d310 con 0x7fe89c0ffd20 2023-07-14T18:12:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8a2526700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c19d450 con 0x7fe89c0feb30 2023-07-14T18:12:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8a2526700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c19d5e0 con 0x7fe89c1006b0 2023-07-14T18:12:08.762 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 0x7fe89c195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:08.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 0x7fe89c195cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54354/0 (socket says 172.21.15.115:54354) 2023-07-14T18:12:08.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89b7fe700 1 -- 172.21.15.115:0/863816575 learned_addr learned my addr 172.21.15.115:0/863816575 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:08.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89bfff700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89c0feb30 0x7fe89c195790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:08.763 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89b7fe700 1 -- 172.21.15.115:0/863816575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe89c1006b0 msgr2=0x7fe89c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89b7fe700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe89c1006b0 0x7fe89c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89b7fe700 1 -- 172.21.15.115:0/863816575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89c0feb30 msgr2=0x7fe89c195790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89b7fe700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89c0feb30 0x7fe89c195790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89b7fe700 1 -- 172.21.15.115:0/863816575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900095f0 con 0x7fe89c0ffd20 2023-07-14T18:12:08.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe89b7fe700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 0x7fe89c195cd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe890005f50 tx=0x7fe890005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8997fa700 1 -- 172.21.15.115:0/863816575 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89001d070 con 0x7fe89c0ffd20 2023-07-14T18:12:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c19d860 con 0x7fe89c0ffd20 2023-07-14T18:12:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8997fa700 1 -- 172.21.15.115:0/863816575 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe890004170 con 0x7fe89c0ffd20 2023-07-14T18:12:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8997fa700 1 -- 172.21.15.115:0/863816575 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890022e60 con 0x7fe89c0ffd20 2023-07-14T18:12:08.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.718+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c19dc90 con 0x7fe89c0ffd20 2023-07-14T18:12:08.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.719+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c10a1b0 con 0x7fe89c0ffd20 2023-07-14T18:12:08.766 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.721+0000 7fe8997fa700 1 -- 172.21.15.115:0/863816575 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fe890022600 con 0x7fe89c0ffd20 2023-07-14T18:12:08.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.721+0000 7fe8997fa700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8840a1220 0x7fe8840a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:08.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.722+0000 7fe8997fa700 1 -- 172.21.15.115:0/863816575 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe8900ce610 con 0x7fe89c0ffd20 2023-07-14T18:12:08.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.722+0000 7fe89bfff700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8840a1220 0x7fe8840a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:08.767 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.722+0000 7fe89bfff700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8840a1220 0x7fe8840a36d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0ffb80 tx=0x7fe88c005cf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:08.768 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.725+0000 7fe8997fa700 1 -- 172.21.15.115:0/863816575 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe890097980 con 0x7fe89c0ffd20 2023-07-14T18:12:08.957 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.917+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe89c19e2d0 con 0x7fe89c0ffd20 2023-07-14T18:12:08.958 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.918+0000 7fe8997fa700 1 -- 172.21.15.115:0/863816575 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24558 (secure 0 0 0) 0x7fe89009afa0 con 0x7fe89c0ffd20 2023-07-14T18:12:08.959 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:12:08.960 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":89,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:11:30.172466+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:12:08.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8840a1220 msgr2=0x7fe8840a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:08.962 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8840a1220 0x7fe8840a36d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0ffb80 tx=0x7fe88c005cf0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 msgr2=0x7fe89c195cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:08.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 0x7fe89c195cd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe890005f50 tx=0x7fe890005740 comp rx=0 tx=0).stop 2023-07-14T18:12:08.963 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 shutdown_connections 2023-07-14T18:12:08.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe8840a1220 0x7fe8840a36d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.964 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe89c1006b0 0x7fe89c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89c0feb30 0x7fe89c195790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.965 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 --2- 172.21.15.115:0/863816575 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe89c0ffd20 0x7fe89c195cd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:08.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.920+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 >> 172.21.15.115:0/863816575 conn(0x7fe89c0fa110 msgr2=0x7fe89c068810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:08.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.921+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 shutdown_connections 2023-07-14T18:12:08.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:08.921+0000 7fe8a2526700 1 -- 172.21.15.115:0/863816575 wait complete. 2023-07-14T18:12:08.975 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:10:15.471213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:10:16.791973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:10:20.663275+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:12:08.976 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:12:09.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.062+0000 7f16728d5700 1 -- 172.21.15.115:0/1992723747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 msgr2=0x7f166c100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.062+0000 7f16728d5700 1 --2- 172.21.15.115:0/1992723747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 0x7f166c100020 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f165c009990 tx=0x7f165c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.105 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.062+0000 7f16728d5700 1 -- 172.21.15.115:0/1992723747 shutdown_connections 2023-07-14T18:12:09.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.062+0000 7f16728d5700 1 --2- 172.21.15.115:0/1992723747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f166c101790 0x7f166c108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.062+0000 7f16728d5700 1 --2- 172.21.15.115:0/1992723747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 0x7f166c100020 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.062+0000 7f16728d5700 1 --2- 172.21.15.115:0/1992723747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f166c100e00 0x7f166c101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.062+0000 7f16728d5700 1 -- 172.21.15.115:0/1992723747 >> 172.21.15.115:0/1992723747 conn(0x7f166c0fb250 msgr2=0x7f166c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:09.106 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.062+0000 7f16728d5700 1 -- 172.21.15.115:0/1992723747 shutdown_connections 2023-07-14T18:12:09.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.062+0000 7f16728d5700 1 -- 172.21.15.115:0/1992723747 wait complete. 2023-07-14T18:12:09.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.063+0000 7f16728d5700 1 Processor -- start 2023-07-14T18:12:09.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.063+0000 7f16728d5700 1 -- start start 2023-07-14T18:12:09.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16728d5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 0x7f166c1958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.107 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16728d5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f166c100e00 0x7f166c195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16728d5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f166c101790 0x7f166c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16728d5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c19d430 con 0x7f166c0ffc10 2023-07-14T18:12:09.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16728d5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c19d570 con 0x7f166c101790 2023-07-14T18:12:09.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16728d5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c19d700 con 0x7f166c100e00 2023-07-14T18:12:09.108 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 0x7f166c1958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 0x7f166c1958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54366/0 (socket says 172.21.15.115:54366) 2023-07-14T18:12:09.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166bfff700 1 -- 172.21.15.115:0/2929161147 learned_addr learned my addr 172.21.15.115:0/2929161147 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:09.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166b7fe700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f166c100e00 0x7f166c195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f1670e72700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f166c101790 0x7f166c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.109 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166bfff700 1 -- 172.21.15.115:0/2929161147 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f166c100e00 msgr2=0x7f166c195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166bfff700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f166c100e00 0x7f166c195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166bfff700 1 -- 172.21.15.115:0/2929161147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f166c101790 msgr2=0x7f166c19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166bfff700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f166c101790 0x7f166c19ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166bfff700 1 -- 172.21.15.115:0/2929161147 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c0095f0 con 0x7f166c0ffc10 2023-07-14T18:12:09.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f166bfff700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 0x7f166c1958b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f165c000c00 tx=0x7f165c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:09.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16697fa700 1 -- 172.21.15.115:0/2929161147 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c01d070 con 0x7f166c0ffc10 2023-07-14T18:12:09.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16697fa700 1 -- 172.21.15.115:0/2929161147 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f165c022470 con 0x7f166c0ffc10 2023-07-14T18:12:09.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c19d980 con 0x7f166c0ffc10 2023-07-14T18:12:09.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.064+0000 7f16697fa700 1 -- 172.21.15.115:0/2929161147 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c00f6c0 con 0x7f166c0ffc10 2023-07-14T18:12:09.111 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.065+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c19dde0 con 0x7f166c0ffc10 2023-07-14T18:12:09.112 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.066+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c10b290 con 0x7f166c0ffc10 2023-07-14T18:12:09.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.066+0000 7f16697fa700 1 -- 172.21.15.115:0/2929161147 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f165c0225e0 con 0x7f166c0ffc10 2023-07-14T18:12:09.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.067+0000 7f16697fa700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16540a11e0 0x7f16540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.067+0000 7f166b7fe700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16540a11e0 0x7f16540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.067+0000 7f16697fa700 1 -- 172.21.15.115:0/2929161147 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f165c0ce240 con 0x7f166c0ffc10 2023-07-14T18:12:09.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.067+0000 7f166b7fe700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16540a11e0 0x7f16540a3690 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1660005950 tx=0x7f166000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:09.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.071+0000 7f16697fa700 1 -- 172.21.15.115:0/2929161147 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f165c0975b0 con 0x7f166c0ffc10 2023-07-14T18:12:09.305 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.265+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f166c19e3c0 con 0x7f166c0ffc10 2023-07-14T18:12:09.306 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.265+0000 7f16697fa700 1 -- 172.21.15.115:0/2929161147 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f165c09abd0 con 0x7f166c0ffc10 2023-07-14T18:12:09.306 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:12:09.307 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16540a11e0 msgr2=0x7f16540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16540a11e0 0x7f16540a3690 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1660005950 tx=0x7f166000a300 comp rx=0 tx=0).stop 2023-07-14T18:12:09.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 msgr2=0x7f166c1958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 0x7f166c1958b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f165c000c00 tx=0x7f165c003820 comp rx=0 tx=0).stop 2023-07-14T18:12:09.308 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 shutdown_connections 2023-07-14T18:12:09.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f16540a11e0 0x7f16540a3690 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f166c101790 0x7f166c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f166c0ffc10 0x7f166c1958b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 --2- 172.21.15.115:0/2929161147 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f166c100e00 0x7f166c195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.309 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 >> 172.21.15.115:0/2929161147 conn(0x7f166c0fb250 msgr2=0x7f166c0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:09.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 shutdown_connections 2023-07-14T18:12:09.310 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.267+0000 7f16728d5700 1 -- 172.21.15.115:0/2929161147 wait complete. 2023-07-14T18:12:09.319 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:09 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/863816575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:12:09.320 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:12:09.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:09 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/863816575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:12:09.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:09 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/863816575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:12:09.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.403+0000 7f88c5e4e700 1 -- 172.21.15.115:0/3065452305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 msgr2=0x7f88c00fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.403+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/3065452305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 0x7f88c00fe770 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f88b800b0a0 tx=0x7f88b800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 -- 172.21.15.115:0/3065452305 shutdown_connections 2023-07-14T18:12:09.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/3065452305 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 0x7f88c00fe770 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/3065452305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88c0101260 0x7f88c0101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.445 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/3065452305 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88c0101bb0 0x7f88c00fdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 -- 172.21.15.115:0/3065452305 >> 172.21.15.115:0/3065452305 conn(0x7f88c00f98a0 msgr2=0x7f88c00fbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:09.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 -- 172.21.15.115:0/3065452305 shutdown_connections 2023-07-14T18:12:09.446 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 -- 172.21.15.115:0/3065452305 wait complete. 2023-07-14T18:12:09.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 Processor -- start 2023-07-14T18:12:09.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 -- start start 2023-07-14T18:12:09.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 0x7f88c01959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88c0101260 0x7f88c0195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88c0101bb0 0x7f88c019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c019d530 con 0x7f88c00fe2f0 2023-07-14T18:12:09.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c019d670 con 0x7f88c0101260 2023-07-14T18:12:09.449 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.404+0000 7f88c5e4e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c019d800 con 0x7f88c0101bb0 2023-07-14T18:12:09.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bf7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 0x7f88c01959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bf7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 0x7f88c01959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54380/0 (socket says 172.21.15.115:54380) 2023-07-14T18:12:09.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bf7fe700 1 -- 172.21.15.115:0/2199792928 learned_addr learned my addr 172.21.15.115:0/2199792928 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:09.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bffff700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88c0101bb0 0x7f88c019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88beffd700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88c0101260 0x7f88c0195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bf7fe700 1 -- 172.21.15.115:0/2199792928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88c0101bb0 msgr2=0x7f88c019cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bf7fe700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88c0101bb0 0x7f88c019cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bf7fe700 1 -- 172.21.15.115:0/2199792928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88c0101260 msgr2=0x7f88c0195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bf7fe700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88c0101260 0x7f88c0195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bf7fe700 1 -- 172.21.15.115:0/2199792928 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b8009d00 con 0x7f88c00fe2f0 2023-07-14T18:12:09.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88beffd700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88c0101260 0x7f88c0195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:12:09.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bf7fe700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 0x7f88c01959b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f88b000d950 tx=0x7f88b000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:09.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bcff9700 1 -- 172.21.15.115:0/2199792928 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b0009940 con 0x7f88c00fe2f0 2023-07-14T18:12:09.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bcff9700 1 -- 172.21.15.115:0/2199792928 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f88b0010460 con 0x7f88c00fe2f0 2023-07-14T18:12:09.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c019dae0 con 0x7f88c00fe2f0 2023-07-14T18:12:09.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88bcff9700 1 -- 172.21.15.115:0/2199792928 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b000f620 con 0x7f88c00fe2f0 2023-07-14T18:12:09.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.405+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c019dff0 con 0x7f88c00fe2f0 2023-07-14T18:12:09.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.406+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c01069d0 con 0x7f88c00fe2f0 2023-07-14T18:12:09.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.407+0000 7f88bcff9700 1 -- 172.21.15.115:0/2199792928 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f88b00105d0 con 0x7f88c00fe2f0 2023-07-14T18:12:09.453 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.408+0000 7f88bcff9700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f88a80a11e0 0x7f88a80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.408+0000 7f88beffd700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f88a80a11e0 0x7f88a80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.408+0000 7f88bcff9700 1 -- 172.21.15.115:0/2199792928 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f88b00ccc20 con 0x7f88c00fe2f0 2023-07-14T18:12:09.454 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.408+0000 7f88beffd700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f88a80a11e0 0x7f88a80a3690 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009cb0 tx=0x7f88b4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:09.455 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.411+0000 7f88bcff9700 1 -- 172.21.15.115:0/2199792928 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f88b00cce60 con 0x7f88c00fe2f0 2023-07-14T18:12:09.638 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.598+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88c002cb50 con 0x7f88c00fe2f0 2023-07-14T18:12:09.639 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.598+0000 7f88bcff9700 1 -- 172.21.15.115:0/2199792928 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88b0016020 con 0x7f88c00fe2f0 2023-07-14T18:12:09.639 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:12:09.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f88a80a11e0 msgr2=0x7f88a80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f88a80a11e0 0x7f88a80a3690 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009cb0 tx=0x7f88b4009260 comp rx=0 tx=0).stop 2023-07-14T18:12:09.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 msgr2=0x7f88c01959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 0x7f88c01959b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f88b000d950 tx=0x7f88b000dc60 comp rx=0 tx=0).stop 2023-07-14T18:12:09.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 shutdown_connections 2023-07-14T18:12:09.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f88a80a11e0 0x7f88a80a3690 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f88c0101bb0 0x7f88c019cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f88c00fe2f0 0x7f88c01959b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 --2- 172.21.15.115:0/2199792928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88c0101260 0x7f88c0195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.600+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 >> 172.21.15.115:0/2199792928 conn(0x7f88c00f98a0 msgr2=0x7f88c00fb620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:09.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.601+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 shutdown_connections 2023-07-14T18:12:09.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.601+0000 7f88c5e4e700 1 -- 172.21.15.115:0/2199792928 wait complete. 2023-07-14T18:12:09.654 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:12:09.780 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.737+0000 7fc443b37700 1 -- 172.21.15.115:0/39859764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c100e20 msgr2=0x7fc43c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.737+0000 7fc443b37700 1 --2- 172.21.15.115:0/39859764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c100e20 0x7fc43c101270 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc438009990 tx=0x7fc438009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.799 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 -- 172.21.15.115:0/39859764 shutdown_connections 2023-07-14T18:12:09.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 --2- 172.21.15.115:0/39859764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c1017b0 0x7fc43c108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.800 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 --2- 172.21.15.115:0/39859764 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc43c0ffc30 0x7fc43c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 --2- 172.21.15.115:0/39859764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c100e20 0x7fc43c101270 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 -- 172.21.15.115:0/39859764 >> 172.21.15.115:0/39859764 conn(0x7fc43c0fb290 msgr2=0x7fc43c0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:09.801 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 -- 172.21.15.115:0/39859764 shutdown_connections 2023-07-14T18:12:09.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 -- 172.21.15.115:0/39859764 wait complete. 2023-07-14T18:12:09.802 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 Processor -- start 2023-07-14T18:12:09.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 -- start start 2023-07-14T18:12:09.803 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c0ffc30 0x7fc43c0743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc43c100e20 0x7fc43c072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.804 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c1017b0 0x7fc43c072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c073710 con 0x7fc43c0ffc30 2023-07-14T18:12:09.805 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c073880 con 0x7fc43c1017b0 2023-07-14T18:12:09.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.738+0000 7fc443b37700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c19f870 con 0x7fc43c100e20 2023-07-14T18:12:09.806 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4418d3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c0ffc30 0x7fc43c0743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4418d3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c0ffc30 0x7fc43c0743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54394/0 (socket says 172.21.15.115:54394) 2023-07-14T18:12:09.807 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4418d3700 1 -- 172.21.15.115:0/4288457845 learned_addr learned my addr 172.21.15.115:0/4288457845 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:09.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4410d2700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc43c100e20 0x7fc43c072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4420d4700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c1017b0 0x7fc43c072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4418d3700 1 -- 172.21.15.115:0/4288457845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc43c100e20 msgr2=0x7fc43c072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.808 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4418d3700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc43c100e20 0x7fc43c072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4418d3700 1 -- 172.21.15.115:0/4288457845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c1017b0 msgr2=0x7fc43c072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4418d3700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c1017b0 0x7fc43c072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4418d3700 1 -- 172.21.15.115:0/4288457845 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4380095f0 con 0x7fc43c0ffc30 2023-07-14T18:12:09.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc4418d3700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c0ffc30 0x7fc43c0743f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00d980 tx=0x7fc42c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:09.809 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc432ffd700 1 -- 172.21.15.115:0/4288457845 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c004020 con 0x7fc43c0ffc30 2023-07-14T18:12:09.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc432ffd700 1 -- 172.21.15.115:0/4288457845 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc42c009730 con 0x7fc43c0ffc30 2023-07-14T18:12:09.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c19fb50 con 0x7fc43c0ffc30 2023-07-14T18:12:09.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc432ffd700 1 -- 172.21.15.115:0/4288457845 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c003a00 con 0x7fc43c0ffc30 2023-07-14T18:12:09.810 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.739+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c1a0060 con 0x7fc43c0ffc30 2023-07-14T18:12:09.811 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.740+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c107210 con 0x7fc43c0ffc30 2023-07-14T18:12:09.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.741+0000 7fc432ffd700 1 -- 172.21.15.115:0/4288457845 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fc42c0098a0 con 0x7fc43c0ffc30 2023-07-14T18:12:09.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.742+0000 7fc432ffd700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc4280a11e0 0x7fc4280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:09.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.742+0000 7fc4410d2700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc4280a11e0 0x7fc4280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:09.812 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.742+0000 7fc432ffd700 1 -- 172.21.15.115:0/4288457845 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc42c0cce90 con 0x7fc43c0ffc30 2023-07-14T18:12:09.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.742+0000 7fc4410d2700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc4280a11e0 0x7fc4280a3690 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c010 tx=0x7fc438005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:09.813 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.745+0000 7fc432ffd700 1 -- 172.21.15.115:0/4288457845 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc42c096110 con 0x7fc43c0ffc30 2023-07-14T18:12:09.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.934+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc43c02cb50 con 0x7fc43c0ffc30 2023-07-14T18:12:09.975 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.934+0000 7fc432ffd700 1 -- 172.21.15.115:0/4288457845 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc42c099730 con 0x7fc43c0ffc30 2023-07-14T18:12:09.976 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 64 2023-07-14T18:12:09.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc4280a11e0 msgr2=0x7fc4280a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc4280a11e0 0x7fc4280a3690 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c010 tx=0x7fc438005960 comp rx=0 tx=0).stop 2023-07-14T18:12:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c0ffc30 msgr2=0x7fc43c0743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c0ffc30 0x7fc43c0743f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00d980 tx=0x7fc42c00dc90 comp rx=0 tx=0).stop 2023-07-14T18:12:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 shutdown_connections 2023-07-14T18:12:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc4280a11e0 0x7fc4280a3690 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc43c1017b0 0x7fc43c072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc43c0ffc30 0x7fc43c0743f0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 --2- 172.21.15.115:0/4288457845 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc43c100e20 0x7fc43c072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 >> 172.21.15.115:0/4288457845 conn(0x7fc43c0fb290 msgr2=0x7fc43c104350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 shutdown_connections 2023-07-14T18:12:09.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:09.936+0000 7fc443b37700 1 -- 172.21.15.115:0/4288457845 wait complete. 2023-07-14T18:12:09.989 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:12:10.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.076+0000 7fdb54ecb700 1 -- 172.21.15.115:0/4265532182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb50100690 msgr2=0x7fdb50106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.076+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/4265532182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb50100690 0x7fdb50106f50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800b0a0 tx=0x7fdb4800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.076+0000 7fdb54ecb700 1 -- 172.21.15.115:0/4265532182 shutdown_connections 2023-07-14T18:12:10.117 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.076+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/4265532182 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb50100690 0x7fdb50106f50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.076+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/4265532182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb500feb10 0x7fdb500fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.076+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/4265532182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb500ffd00 0x7fdb50100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.076+0000 7fdb54ecb700 1 -- 172.21.15.115:0/4265532182 >> 172.21.15.115:0/4265532182 conn(0x7fdb500fa130 msgr2=0x7fdb500fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:10.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.076+0000 7fdb54ecb700 1 -- 172.21.15.115:0/4265532182 shutdown_connections 2023-07-14T18:12:10.118 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.076+0000 7fdb54ecb700 1 -- 172.21.15.115:0/4265532182 wait complete. 2023-07-14T18:12:10.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb54ecb700 1 Processor -- start 2023-07-14T18:12:10.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb54ecb700 1 -- start start 2023-07-14T18:12:10.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb54ecb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb500feb10 0x7fdb50195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb54ecb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb500ffd00 0x7fdb50195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb54ecb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb50100690 0x7fdb5019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb54ecb700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5019d510 con 0x7fdb500ffd00 2023-07-14T18:12:10.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb54ecb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5019d650 con 0x7fdb500feb10 2023-07-14T18:12:10.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb54ecb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5019d7e0 con 0x7fdb50100690 2023-07-14T18:12:10.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb500ffd00 0x7fdb50195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4dd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb500ffd00 0x7fdb50195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54410/0 (socket says 172.21.15.115:54410) 2023-07-14T18:12:10.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4dd9b700 1 -- 172.21.15.115:0/3486886287 learned_addr learned my addr 172.21.15.115:0/3486886287 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:10.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4e59c700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb500feb10 0x7fdb50195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4ed9d700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb50100690 0x7fdb5019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4dd9b700 1 -- 172.21.15.115:0/3486886287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb50100690 msgr2=0x7fdb5019cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4dd9b700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb50100690 0x7fdb5019cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.123 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4dd9b700 1 -- 172.21.15.115:0/3486886287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb500feb10 msgr2=0x7fdb50195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4dd9b700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb500feb10 0x7fdb50195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4dd9b700 1 -- 172.21.15.115:0/3486886287 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb48009d00 con 0x7fdb500ffd00 2023-07-14T18:12:10.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb4dd9b700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb500ffd00 0x7fdb50195ed0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000c920 tx=0x7fdb4000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:10.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.077+0000 7fdb3f7fe700 1 -- 172.21.15.115:0/3486886287 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40007b00 con 0x7fdb500ffd00 2023-07-14T18:12:10.124 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.078+0000 7fdb3f7fe700 1 -- 172.21.15.115:0/3486886287 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdb4000ce70 con 0x7fdb500ffd00 2023-07-14T18:12:10.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.078+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5019dac0 con 0x7fdb500ffd00 2023-07-14T18:12:10.125 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.078+0000 7fdb3f7fe700 1 -- 172.21.15.115:0/3486886287 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40005440 con 0x7fdb500ffd00 2023-07-14T18:12:10.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.078+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5019dfd0 con 0x7fdb500ffd00 2023-07-14T18:12:10.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.078+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb50105a60 con 0x7fdb500ffd00 2023-07-14T18:12:10.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.080+0000 7fdb3f7fe700 1 -- 172.21.15.115:0/3486886287 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fdb400055a0 con 0x7fdb500ffd00 2023-07-14T18:12:10.126 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.080+0000 7fdb3f7fe700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdb380a11e0 0x7fdb380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.080+0000 7fdb3f7fe700 1 -- 172.21.15.115:0/3486886287 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdb400ccdc0 con 0x7fdb500ffd00 2023-07-14T18:12:10.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.080+0000 7fdb4e59c700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdb380a11e0 0x7fdb380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.081+0000 7fdb4e59c700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdb380a11e0 0x7fdb380a3690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdb500ffb60 tx=0x7fdb4400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:10.127 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.084+0000 7fdb3f7fe700 1 -- 172.21.15.115:0/3486886287 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdb400960a0 con 0x7fdb500ffd00 2023-07-14T18:12:10.315 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.275+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdb5002caf0 con 0x7fdb500ffd00 2023-07-14T18:12:10.316 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.275+0000 7fdb3f7fe700 1 -- 172.21.15.115:0/3486886287 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdb4001d020 con 0x7fdb500ffd00 2023-07-14T18:12:10.317 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:12:10.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.277+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdb380a11e0 msgr2=0x7fdb380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.318 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.277+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdb380a11e0 0x7fdb380a3690 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdb500ffb60 tx=0x7fdb4400b410 comp rx=0 tx=0).stop 2023-07-14T18:12:10.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.277+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb500ffd00 msgr2=0x7fdb50195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.277+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb500ffd00 0x7fdb50195ed0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000c920 tx=0x7fdb4000cc30 comp rx=0 tx=0).stop 2023-07-14T18:12:10.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.278+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 shutdown_connections 2023-07-14T18:12:10.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.278+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdb380a11e0 0x7fdb380a3690 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.319 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.278+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb50100690 0x7fdb5019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.278+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb500feb10 0x7fdb50195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.278+0000 7fdb54ecb700 1 --2- 172.21.15.115:0/3486886287 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdb500ffd00 0x7fdb50195ed0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.278+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 >> 172.21.15.115:0/3486886287 conn(0x7fdb500fa130 msgr2=0x7fdb500fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:10.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.278+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 shutdown_connections 2023-07-14T18:12:10.320 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.278+0000 7fdb54ecb700 1 -- 172.21.15.115:0/3486886287 wait complete. 2023-07-14T18:12:10.329 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:10 smithi115 ceph-mon[127451]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-07-14T18:12:10.329 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:10 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2929161147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:12:10.330 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:10 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2199792928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:12:10.330 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:10 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/4288457845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:12:10.331 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T18:12:10.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:10 smithi150 ceph-mon[128853]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-07-14T18:12:10.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:10 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2929161147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:12:10.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:10 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2199792928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:12:10.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:10 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/4288457845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:12:10.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:10 smithi187 ceph-mon[129317]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-07-14T18:12:10.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:10 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2929161147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:12:10.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:10 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2199792928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:12:10.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:10 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/4288457845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:12:10.456 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.415+0000 7f35e344e700 1 -- 172.21.15.115:0/1547119886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 msgr2=0x7f35dc107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.415+0000 7f35e344e700 1 --2- 172.21.15.115:0/1547119886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 0x7f35dc107010 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f35d400d3a0 tx=0x7f35d400d6b0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.415+0000 7f35e344e700 1 -- 172.21.15.115:0/1547119886 shutdown_connections 2023-07-14T18:12:10.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.415+0000 7f35e344e700 1 --2- 172.21.15.115:0/1547119886 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 0x7f35dc107010 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.457 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.415+0000 7f35e344e700 1 --2- 172.21.15.115:0/1547119886 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35dc0feb30 0x7f35dc0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.415+0000 7f35e344e700 1 --2- 172.21.15.115:0/1547119886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35dc0ffdc0 0x7f35dc100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.415+0000 7f35e344e700 1 -- 172.21.15.115:0/1547119886 >> 172.21.15.115:0/1547119886 conn(0x7f35dc0fa150 msgr2=0x7f35dc0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:10.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 -- 172.21.15.115:0/1547119886 shutdown_connections 2023-07-14T18:12:10.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 -- 172.21.15.115:0/1547119886 wait complete. 2023-07-14T18:12:10.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 Processor -- start 2023-07-14T18:12:10.459 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 -- start start 2023-07-14T18:12:10.460 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35dc0feb30 0x7f35dc1935e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35dc0ffdc0 0x7f35dc193b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 0x7f35dc19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc19b160 con 0x7f35dc100770 2023-07-14T18:12:10.461 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc19b2a0 con 0x7f35dc0ffdc0 2023-07-14T18:12:10.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e344e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc19b430 con 0x7f35dc0feb30 2023-07-14T18:12:10.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e19eb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 0x7f35dc19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e19eb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 0x7f35dc19ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54426/0 (socket says 172.21.15.115:54426) 2023-07-14T18:12:10.462 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.416+0000 7f35e19eb700 1 -- 172.21.15.115:0/2485382610 learned_addr learned my addr 172.21.15.115:0/2485382610 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:10.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e09e9700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35dc0ffdc0 0x7f35dc193b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e11ea700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35dc0feb30 0x7f35dc1935e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e19eb700 1 -- 172.21.15.115:0/2485382610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35dc0feb30 msgr2=0x7f35dc1935e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e19eb700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35dc0feb30 0x7f35dc1935e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e19eb700 1 -- 172.21.15.115:0/2485382610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35dc0ffdc0 msgr2=0x7f35dc193b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.463 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e19eb700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35dc0ffdc0 0x7f35dc193b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e19eb700 1 -- 172.21.15.115:0/2485382610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d400d050 con 0x7f35dc100770 2023-07-14T18:12:10.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e11ea700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35dc0feb30 0x7f35dc1935e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:12:10.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e19eb700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 0x7f35dc19ab90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f35d400b780 tx=0x7f35d400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:10.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35d27fc700 1 -- 172.21.15.115:0/2485382610 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4010040 con 0x7f35dc100770 2023-07-14T18:12:10.464 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35d27fc700 1 -- 172.21.15.115:0/2485382610 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f35d401fde0 con 0x7f35dc100770 2023-07-14T18:12:10.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35d27fc700 1 -- 172.21.15.115:0/2485382610 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4014de0 con 0x7f35dc100770 2023-07-14T18:12:10.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35dc19b6b0 con 0x7f35dc100770 2023-07-14T18:12:10.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.417+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35dc19bae0 con 0x7f35dc100770 2023-07-14T18:12:10.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.418+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35dc109160 con 0x7f35dc100770 2023-07-14T18:12:10.465 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.419+0000 7f35d27fc700 1 -- 172.21.15.115:0/2485382610 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f35d4014570 con 0x7f35dc100770 2023-07-14T18:12:10.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.420+0000 7f35d27fc700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35c80a11e0 0x7f35c80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.420+0000 7f35e11ea700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35c80a11e0 0x7f35c80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.420+0000 7f35d27fc700 1 -- 172.21.15.115:0/2485382610 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f35d40d1120 con 0x7f35dc100770 2023-07-14T18:12:10.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.420+0000 7f35e11ea700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35c80a11e0 0x7f35c80a3690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f35d8009be0 tx=0x7f35d8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:10.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.423+0000 7f35d27fc700 1 -- 172.21.15.115:0/2485382610 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f35d409a490 con 0x7f35dc100770 2023-07-14T18:12:10.670 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.629+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f35dc105a80 con 0x7f35dc100770 2023-07-14T18:12:10.671 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.630+0000 7f35d27fc700 1 -- 172.21.15.115:0/2485382610 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+600 (secure 0 0 0) 0x7f35d4023270 con 0x7f35dc100770 2023-07-14T18:12:10.671 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6800/2169355495 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:10.671 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/4031581902 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:10.672 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/3311800921 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:10.672 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6801/2169355495 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:10.672 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2790161712 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:10.672 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2625519889 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:10.673 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6801/568655687 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:10.673 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/3340901626 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:10.673 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6800/568655687 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:10.673 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2416566340 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:10.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35c80a11e0 msgr2=0x7f35c80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35c80a11e0 0x7f35c80a3690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f35d8009be0 tx=0x7f35d8009260 comp rx=0 tx=0).stop 2023-07-14T18:12:10.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 msgr2=0x7f35dc19ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 0x7f35dc19ab90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f35d400b780 tx=0x7f35d400be00 comp rx=0 tx=0).stop 2023-07-14T18:12:10.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 shutdown_connections 2023-07-14T18:12:10.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f35c80a11e0 0x7f35c80a3690 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35dc100770 0x7f35dc19ab90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f35dc0feb30 0x7f35dc1935e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 --2- 172.21.15.115:0/2485382610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35dc0ffdc0 0x7f35dc193b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 >> 172.21.15.115:0/2485382610 conn(0x7f35dc0fa150 msgr2=0x7f35dc0fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:10.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 shutdown_connections 2023-07-14T18:12:10.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.632+0000 7f35e344e700 1 -- 172.21.15.115:0/2485382610 wait complete. 2023-07-14T18:12:10.676 INFO:teuthology.orchestra.run.smithi115.stderr:listed 10 entries 2023-07-14T18:12:10.684 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-07-14T18:12:10.685 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:12:10.703 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:12:10.864 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.823+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3015090793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc0100680 msgr2=0x7fcbc0106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.823+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3015090793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc0100680 0x7fcbc0106f40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800b0a0 tx=0x7fcbb800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.823+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3015090793 shutdown_connections 2023-07-14T18:12:10.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.823+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3015090793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc0100680 0x7fcbc0106f40 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.823+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3015090793 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc00feb00 0x7fcbc00fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.865 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.823+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3015090793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc00ffcf0 0x7fcbc0100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.823+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3015090793 >> 172.21.15.115:0/3015090793 conn(0x7fcbc00fa100 msgr2=0x7fcbc00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:10.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.823+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3015090793 shutdown_connections 2023-07-14T18:12:10.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.823+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3015090793 wait complete. 2023-07-14T18:12:10.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc67fb700 1 Processor -- start 2023-07-14T18:12:10.866 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc67fb700 1 -- start start 2023-07-14T18:12:10.868 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc67fb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc00feb00 0x7fcbc0068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc67fb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc00ffcf0 0x7fcbc019a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc67fb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc0100680 0x7fcbc019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc67fb700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc019d440 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc67fb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc019d5b0 con 0x7fcbc00feb00 2023-07-14T18:12:10.869 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc67fb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc019d770 con 0x7fcbc0100680 2023-07-14T18:12:10.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbb7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc00ffcf0 0x7fcbc019a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbb7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc00ffcf0 0x7fcbc019a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54438/0 (socket says 172.21.15.115:54438) 2023-07-14T18:12:10.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbb7fff700 1 -- 172.21.15.115:0/3775197667 learned_addr learned my addr 172.21.15.115:0/3775197667 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:10.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbbffff700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc00feb00 0x7fcbc0068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.870 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc4d98700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc0100680 0x7fcbc019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbb7fff700 1 -- 172.21.15.115:0/3775197667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc0100680 msgr2=0x7fcbc019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbb7fff700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc0100680 0x7fcbc019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbb7fff700 1 -- 172.21.15.115:0/3775197667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc00feb00 msgr2=0x7fcbc0068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:10.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbb7fff700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc00feb00 0x7fcbc0068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:10.871 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbb7fff700 1 -- 172.21.15.115:0/3775197667 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb8009d00 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbc4d98700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc0100680 0x7fcbc019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:12:10.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.824+0000 7fcbb7fff700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc00ffcf0 0x7fcbc019a620 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcba800d950 tx=0x7fcba800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:10.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.825+0000 7fcbbdffb700 1 -- 172.21.15.115:0/3775197667 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba8009940 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.825+0000 7fcbbdffb700 1 -- 172.21.15.115:0/3775197667 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcba8010460 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.872 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.825+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc019da50 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.825+0000 7fcbbdffb700 1 -- 172.21.15.115:0/3775197667 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba800f620 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.825+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc019df60 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.826+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc010a200 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.827+0000 7fcbbdffb700 1 -- 172.21.15.115:0/3775197667 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fcba80105d0 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.873 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.827+0000 7fcbbdffb700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbac0a1220 0x7fcbac0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:10.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.827+0000 7fcbbdffb700 1 -- 172.21.15.115:0/3775197667 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcba80ccc10 con 0x7fcbc00ffcf0 2023-07-14T18:12:10.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.827+0000 7fcbbffff700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbac0a1220 0x7fcbac0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:10.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.828+0000 7fcbbffff700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbac0a1220 0x7fcbac0a36d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0005950 tx=0x7fcbb000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:10.874 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:10.830+0000 7fcbbdffb700 1 -- 172.21.15.115:0/3775197667 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcba8095ef0 con 0x7fcbc00ffcf0 2023-07-14T18:12:11.064 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.024+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbc0105990 con 0x7fcbc00ffcf0 2023-07-14T18:12:11.065 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.025+0000 7fcbbdffb700 1 -- 172.21.15.115:0/3775197667 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24558 (secure 0 0 0) 0x7fcba8020090 con 0x7fcbc00ffcf0 2023-07-14T18:12:11.066 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:12:11.066 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":89,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:11:30.172466+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:12:11.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbac0a1220 msgr2=0x7fcbac0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbac0a1220 0x7fcbac0a36d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0005950 tx=0x7fcbb000b410 comp rx=0 tx=0).stop 2023-07-14T18:12:11.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc00ffcf0 msgr2=0x7fcbc019a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.068 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc00ffcf0 0x7fcbc019a620 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcba800d950 tx=0x7fcba800dc60 comp rx=0 tx=0).stop 2023-07-14T18:12:11.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 shutdown_connections 2023-07-14T18:12:11.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fcbac0a1220 0x7fcbac0a36d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcbc0100680 0x7fcbc019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc00feb00 0x7fcbc0068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.069 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 --2- 172.21.15.115:0/3775197667 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcbc00ffcf0 0x7fcbc019a620 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 >> 172.21.15.115:0/3775197667 conn(0x7fcbc00fa100 msgr2=0x7fcbc00fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:11.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 shutdown_connections 2023-07-14T18:12:11.070 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.027+0000 7fcbc67fb700 1 -- 172.21.15.115:0/3775197667 wait complete. 2023-07-14T18:12:11.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:06:31.134937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:10:15.471213+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:10:16.791973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:10:20.663275+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:12:11.081 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:12:11.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.165+0000 7f4417bc3700 1 -- 172.21.15.115:0/175819301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410100e00 msgr2=0x7f4410101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.206 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.165+0000 7f4417bc3700 1 --2- 172.21.15.115:0/175819301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410100e00 0x7f4410101250 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4400009990 tx=0x7f4400009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.165+0000 7f4417bc3700 1 -- 172.21.15.115:0/175819301 shutdown_connections 2023-07-14T18:12:11.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.165+0000 7f4417bc3700 1 --2- 172.21.15.115:0/175819301 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4410101790 0x7f4410108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.165+0000 7f4417bc3700 1 --2- 172.21.15.115:0/175819301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44100ffc10 0x7f4410100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.207 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.165+0000 7f4417bc3700 1 --2- 172.21.15.115:0/175819301 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410100e00 0x7f4410101250 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.165+0000 7f4417bc3700 1 -- 172.21.15.115:0/175819301 >> 172.21.15.115:0/175819301 conn(0x7f44100fb250 msgr2=0x7f44100fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:11.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.165+0000 7f4417bc3700 1 -- 172.21.15.115:0/175819301 shutdown_connections 2023-07-14T18:12:11.208 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.165+0000 7f4417bc3700 1 -- 172.21.15.115:0/175819301 wait complete. 2023-07-14T18:12:11.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4417bc3700 1 Processor -- start 2023-07-14T18:12:11.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4417bc3700 1 -- start start 2023-07-14T18:12:11.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4417bc3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44100ffc10 0x7f44101957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.210 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4417bc3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4410100e00 0x7f4410195d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4417bc3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410101790 0x7f441019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4417bc3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441019d340 con 0x7f4410101790 2023-07-14T18:12:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4417bc3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441019d480 con 0x7f44100ffc10 2023-07-14T18:12:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4417bc3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441019d610 con 0x7f4410100e00 2023-07-14T18:12:11.211 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4416160700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410101790 0x7f441019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4416160700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410101790 0x7f441019cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54454/0 (socket says 172.21.15.115:54454) 2023-07-14T18:12:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f4416160700 1 -- 172.21.15.115:0/1722676853 learned_addr learned my addr 172.21.15.115:0/1722676853 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f441595f700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44100ffc10 0x7f44101957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.166+0000 7f441515e700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4410100e00 0x7f4410195d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4416160700 1 -- 172.21.15.115:0/1722676853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4410100e00 msgr2=0x7f4410195d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4416160700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4410100e00 0x7f4410195d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4416160700 1 -- 172.21.15.115:0/1722676853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44100ffc10 msgr2=0x7f44101957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4416160700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44100ffc10 0x7f44101957c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4416160700 1 -- 172.21.15.115:0/1722676853 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000095f0 con 0x7f4410101790 2023-07-14T18:12:11.213 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4416160700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410101790 0x7f441019cd70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f440800ef30 tx=0x7f440800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:11.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4406ffd700 1 -- 172.21.15.115:0/1722676853 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408018070 con 0x7f4410101790 2023-07-14T18:12:11.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4406ffd700 1 -- 172.21.15.115:0/1722676853 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f440800f040 con 0x7f4410101790 2023-07-14T18:12:11.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4406ffd700 1 -- 172.21.15.115:0/1722676853 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440800cbc0 con 0x7f4410101790 2023-07-14T18:12:11.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441019d890 con 0x7f4410101790 2023-07-14T18:12:11.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.167+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441019dd50 con 0x7f4410101790 2023-07-14T18:12:11.214 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.168+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4410106b60 con 0x7f4410101790 2023-07-14T18:12:11.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.170+0000 7f4406ffd700 1 -- 172.21.15.115:0/1722676853 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f440800cd80 con 0x7f4410101790 2023-07-14T18:12:11.215 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.170+0000 7f4406ffd700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f43fc0a1260 0x7f43fc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.170+0000 7f441595f700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f43fc0a1260 0x7f43fc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.170+0000 7f4406ffd700 1 -- 172.21.15.115:0/1722676853 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f44080cce50 con 0x7f4410101790 2023-07-14T18:12:11.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.171+0000 7f441595f700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f43fc0a1260 0x7f43fc0a3710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4410100c60 tx=0x7f440c00a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:11.216 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.174+0000 7f4406ffd700 1 -- 172.21.15.115:0/1722676853 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4408096160 con 0x7f4410101790 2023-07-14T18:12:11.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:11 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3486886287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:12:11.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:11 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2485382610' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:12:11.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:11 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3486886287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:12:11.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:11 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2485382610' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:12:11.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.364+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f441002ca70 con 0x7f4410101790 2023-07-14T18:12:11.406 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.365+0000 7f4406ffd700 1 -- 172.21.15.115:0/1722676853 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4408099780 con 0x7f4410101790 2023-07-14T18:12:11.406 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-07-14T18:12:11.407 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f43fc0a1260 msgr2=0x7f43fc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f43fc0a1260 0x7f43fc0a3710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4410100c60 tx=0x7f440c00a3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410101790 msgr2=0x7f441019cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.408 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410101790 0x7f441019cd70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f440800ef30 tx=0x7f440800c3a0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 shutdown_connections 2023-07-14T18:12:11.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f43fc0a1260 0x7f43fc0a3710 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.409 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4410101790 0x7f441019cd70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44100ffc10 0x7f44101957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 --2- 172.21.15.115:0/1722676853 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4410100e00 0x7f4410195d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 >> 172.21.15.115:0/1722676853 conn(0x7f44100fb250 msgr2=0x7f44100fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:11.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 shutdown_connections 2023-07-14T18:12:11.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.367+0000 7f4417bc3700 1 -- 172.21.15.115:0/1722676853 wait complete. 2023-07-14T18:12:11.419 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:12:11.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:11 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3486886287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:12:11.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:11 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2485382610' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:12:11.543 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 -- 172.21.15.115:0/4188597668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1078102fb0 msgr2=0x7f1078105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 --2- 172.21.15.115:0/4188597668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1078102fb0 0x7f1078105390 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f106c009990 tx=0x7f106c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.544 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 -- 172.21.15.115:0/4188597668 shutdown_connections 2023-07-14T18:12:11.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 --2- 172.21.15.115:0/4188597668 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10781058d0 0x7f1078107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 --2- 172.21.15.115:0/4188597668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1078100690 0x7f1078102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 --2- 172.21.15.115:0/4188597668 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1078102fb0 0x7f1078105390 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.545 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 -- 172.21.15.115:0/4188597668 >> 172.21.15.115:0/4188597668 conn(0x7f10780fa130 msgr2=0x7f10780fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:11.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 -- 172.21.15.115:0/4188597668 shutdown_connections 2023-07-14T18:12:11.546 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 -- 172.21.15.115:0/4188597668 wait complete. 2023-07-14T18:12:11.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.503+0000 7f107ff7c700 1 Processor -- start 2023-07-14T18:12:11.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107ff7c700 1 -- start start 2023-07-14T18:12:11.548 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107ff7c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1078100690 0x7f1078195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107ff7c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1078102fb0 0x7f1078195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107ff7c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10781058d0 0x7f107819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107ff7c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107819d400 con 0x7f10781058d0 2023-07-14T18:12:11.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107ff7c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107819d540 con 0x7f1078100690 2023-07-14T18:12:11.549 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107ff7c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107819d6d0 con 0x7f1078102fb0 2023-07-14T18:12:11.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107e519700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10781058d0 0x7f107819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107e519700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10781058d0 0x7f107819ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54468/0 (socket says 172.21.15.115:54468) 2023-07-14T18:12:11.550 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107e519700 1 -- 172.21.15.115:0/2545342719 learned_addr learned my addr 172.21.15.115:0/2545342719 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:11.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107dd18700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1078100690 0x7f1078195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107d517700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1078102fb0 0x7f1078195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107e519700 1 -- 172.21.15.115:0/2545342719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1078102fb0 msgr2=0x7f1078195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107e519700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1078102fb0 0x7f1078195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.551 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107e519700 1 -- 172.21.15.115:0/2545342719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1078100690 msgr2=0x7f1078195880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107e519700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1078100690 0x7f1078195880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107e519700 1 -- 172.21.15.115:0/2545342719 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c0095f0 con 0x7f10781058d0 2023-07-14T18:12:11.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107dd18700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1078100690 0x7f1078195880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:12:11.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107e519700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10781058d0 0x7f107819ce30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f107000ada0 tx=0x7f107000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:11.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f106affd700 1 -- 172.21.15.115:0/2545342719 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107000f070 con 0x7f10781058d0 2023-07-14T18:12:11.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f106affd700 1 -- 172.21.15.115:0/2545342719 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1070007b30 con 0x7f10781058d0 2023-07-14T18:12:11.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107819d9b0 con 0x7f10781058d0 2023-07-14T18:12:11.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.504+0000 7f106affd700 1 -- 172.21.15.115:0/2545342719 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10700053e0 con 0x7f10781058d0 2023-07-14T18:12:11.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.505+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107819de70 con 0x7f10781058d0 2023-07-14T18:12:11.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.506+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10780fbca0 con 0x7f10781058d0 2023-07-14T18:12:11.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.507+0000 7f106affd700 1 -- 172.21.15.115:0/2545342719 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f107000c980 con 0x7f10781058d0 2023-07-14T18:12:11.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.507+0000 7f106affd700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a11e0 0x7f10640a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.507+0000 7f107dd18700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a11e0 0x7f10640a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.507+0000 7f106affd700 1 -- 172.21.15.115:0/2545342719 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f10700ccad0 con 0x7f10781058d0 2023-07-14T18:12:11.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.508+0000 7f107dd18700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a11e0 0x7f10640a3690 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1074005950 tx=0x7f10740058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:11.555 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.511+0000 7f106affd700 1 -- 172.21.15.115:0/2545342719 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1070095e40 con 0x7f10781058d0 2023-07-14T18:12:11.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.699+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f107802caf0 con 0x7f10781058d0 2023-07-14T18:12:11.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.700+0000 7f106affd700 1 -- 172.21.15.115:0/2545342719 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f107000cc30 con 0x7f10781058d0 2023-07-14T18:12:11.741 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:12:11.742 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a11e0 msgr2=0x7f10640a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a11e0 0x7f10640a3690 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1074005950 tx=0x7f10740058e0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10781058d0 msgr2=0x7f107819ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10781058d0 0x7f107819ce30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f107000ada0 tx=0x7f107000c330 comp rx=0 tx=0).stop 2023-07-14T18:12:11.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 shutdown_connections 2023-07-14T18:12:11.743 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f10640a11e0 0x7f10640a3690 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f10781058d0 0x7f107819ce30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1078100690 0x7f1078195880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 --2- 172.21.15.115:0/2545342719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1078102fb0 0x7f1078195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.744 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 >> 172.21.15.115:0/2545342719 conn(0x7f10780fa130 msgr2=0x7f10781012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:11.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 shutdown_connections 2023-07-14T18:12:11.745 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.702+0000 7f107ff7c700 1 -- 172.21.15.115:0/2545342719 wait complete. 2023-07-14T18:12:11.755 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:12:11.879 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 -- 172.21.15.115:0/424665342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc100690 msgr2=0x7f49bc106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 --2- 172.21.15.115:0/424665342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc100690 0x7f49bc106f50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f49b400b0a0 tx=0x7f49b400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 -- 172.21.15.115:0/424665342 shutdown_connections 2023-07-14T18:12:11.880 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 --2- 172.21.15.115:0/424665342 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc100690 0x7f49bc106f50 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 --2- 172.21.15.115:0/424665342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49bc0feb10 0x7f49bc0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 --2- 172.21.15.115:0/424665342 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49bc0ffd00 0x7f49bc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 -- 172.21.15.115:0/424665342 >> 172.21.15.115:0/424665342 conn(0x7f49bc0fa130 msgr2=0x7f49bc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:11.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 -- 172.21.15.115:0/424665342 shutdown_connections 2023-07-14T18:12:11.881 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 -- 172.21.15.115:0/424665342 wait complete. 2023-07-14T18:12:11.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 Processor -- start 2023-07-14T18:12:11.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.839+0000 7f49c281c700 1 -- start start 2023-07-14T18:12:11.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49c281c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49bc0feb10 0x7f49bc1948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.884 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49c281c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc0ffd00 0x7f49bc194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49c281c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49bc100690 0x7f49bc19be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49c281c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc19c420 con 0x7f49bc0ffd00 2023-07-14T18:12:11.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49c281c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc19c560 con 0x7f49bc0feb10 2023-07-14T18:12:11.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49c281c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc19c6f0 con 0x7f49bc100690 2023-07-14T18:12:11.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc0ffd00 0x7f49bc194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.885 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc0ffd00 0x7f49bc194de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54480/0 (socket says 172.21.15.115:54480) 2023-07-14T18:12:11.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b3fff700 1 -- 172.21.15.115:0/1646102224 learned_addr learned my addr 172.21.15.115:0/1646102224 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:11.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49bbfff700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49bc0feb10 0x7f49bc1948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49c0db9700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49bc100690 0x7f49bc19be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.886 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b3fff700 1 -- 172.21.15.115:0/1646102224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49bc100690 msgr2=0x7f49bc19be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b3fff700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49bc100690 0x7f49bc19be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.887 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b3fff700 1 -- 172.21.15.115:0/1646102224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49bc0feb10 msgr2=0x7f49bc1948a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:11.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b3fff700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49bc0feb10 0x7f49bc1948a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:11.888 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b3fff700 1 -- 172.21.15.115:0/1646102224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b4009d00 con 0x7f49bc0ffd00 2023-07-14T18:12:11.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b3fff700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc0ffd00 0x7f49bc194de0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009970 tx=0x7f49a4009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:11.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b9ffb700 1 -- 172.21.15.115:0/1646102224 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4012070 con 0x7f49bc0ffd00 2023-07-14T18:12:11.889 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b9ffb700 1 -- 172.21.15.115:0/1646102224 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f49a400d260 con 0x7f49bc0ffd00 2023-07-14T18:12:11.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49b9ffb700 1 -- 172.21.15.115:0/1646102224 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4017930 con 0x7f49bc0ffd00 2023-07-14T18:12:11.890 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49bc067800 con 0x7f49bc0ffd00 2023-07-14T18:12:11.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.840+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49bc067d10 con 0x7f49bc0ffd00 2023-07-14T18:12:11.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.841+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49bc10a1a0 con 0x7f49bc0ffd00 2023-07-14T18:12:11.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.842+0000 7f49b9ffb700 1 -- 172.21.15.115:0/1646102224 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f49a4017a90 con 0x7f49bc0ffd00 2023-07-14T18:12:11.891 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.843+0000 7f49b9ffb700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f49a80a1260 0x7f49a80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:11.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.843+0000 7f49bbfff700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f49a80a1260 0x7f49a80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:11.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.843+0000 7f49b9ffb700 1 -- 172.21.15.115:0/1646102224 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f49a40ce0e0 con 0x7f49bc0ffd00 2023-07-14T18:12:11.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.843+0000 7f49bbfff700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f49a80a1260 0x7f49a80a3710 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f49ac005950 tx=0x7f49ac0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:11.893 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:11.846+0000 7f49b9ffb700 1 -- 172.21.15.115:0/1646102224 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f49a40973c0 con 0x7f49bc0ffd00 2023-07-14T18:12:12.079 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.039+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f49bc105f00 con 0x7f49bc0ffd00 2023-07-14T18:12:12.080 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.039+0000 7f49b9ffb700 1 -- 172.21.15.115:0/1646102224 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f49a401c0a0 con 0x7f49bc0ffd00 2023-07-14T18:12:12.081 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 64 2023-07-14T18:12:12.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f49a80a1260 msgr2=0x7f49a80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f49a80a1260 0x7f49a80a3710 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f49ac005950 tx=0x7f49ac0058e0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.082 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc0ffd00 msgr2=0x7f49bc194de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc0ffd00 0x7f49bc194de0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009970 tx=0x7f49a4009d30 comp rx=0 tx=0).stop 2023-07-14T18:12:12.083 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 shutdown_connections 2023-07-14T18:12:12.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f49a80a1260 0x7f49a80a3710 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f49bc100690 0x7f49bc19be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.084 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49bc0feb10 0x7f49bc1948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 --2- 172.21.15.115:0/1646102224 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49bc0ffd00 0x7f49bc194de0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 >> 172.21.15.115:0/1646102224 conn(0x7f49bc0fa130 msgr2=0x7f49bc103240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:12.085 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 shutdown_connections 2023-07-14T18:12:12.086 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.041+0000 7f49c281c700 1 -- 172.21.15.115:0/1646102224 wait complete. 2023-07-14T18:12:12.094 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:12:12.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.177+0000 7f1d43971700 1 -- 172.21.15.115:0/4211101165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c0ff750 msgr2=0x7f1d3c107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.218 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.177+0000 7f1d43971700 1 --2- 172.21.15.115:0/4211101165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c0ff750 0x7f1d3c107020 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1d3400b0a0 tx=0x7f1d3400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.177+0000 7f1d43971700 1 -- 172.21.15.115:0/4211101165 shutdown_connections 2023-07-14T18:12:12.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.177+0000 7f1d43971700 1 --2- 172.21.15.115:0/4211101165 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c0ff750 0x7f1d3c107020 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.177+0000 7f1d43971700 1 --2- 172.21.15.115:0/4211101165 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d3c100090 0x7f1d3c1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.177+0000 7f1d43971700 1 --2- 172.21.15.115:0/4211101165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d3c0fedc0 0x7f1d3c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.219 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.177+0000 7f1d43971700 1 -- 172.21.15.115:0/4211101165 >> 172.21.15.115:0/4211101165 conn(0x7f1d3c0f98a0 msgr2=0x7f1d3c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:12.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.177+0000 7f1d43971700 1 -- 172.21.15.115:0/4211101165 shutdown_connections 2023-07-14T18:12:12.220 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.177+0000 7f1d43971700 1 -- 172.21.15.115:0/4211101165 wait complete. 2023-07-14T18:12:12.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d43971700 1 Processor -- start 2023-07-14T18:12:12.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d43971700 1 -- start start 2023-07-14T18:12:12.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d43971700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d3c0fedc0 0x7f1d3c0fe840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:12.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d43971700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d3c0ff750 0x7f1d3c19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:12.222 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d43971700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c100090 0x7f1d3c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:12.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d43971700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c19d480 con 0x7f1d3c100090 2023-07-14T18:12:12.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d43971700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c19d5f0 con 0x7f1d3c0fedc0 2023-07-14T18:12:12.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d43971700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c19d7b0 con 0x7f1d3c0ff750 2023-07-14T18:12:12.223 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d41f0e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c100090 0x7f1d3c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:12.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d41f0e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c100090 0x7f1d3c19ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54496/0 (socket says 172.21.15.115:54496) 2023-07-14T18:12:12.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d41f0e700 1 -- 172.21.15.115:0/2810817334 learned_addr learned my addr 172.21.15.115:0/2810817334 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:12.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d40f0c700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d3c0ff750 0x7f1d3c19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:12.224 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d4170d700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d3c0fedc0 0x7f1d3c0fe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:12.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d41f0e700 1 -- 172.21.15.115:0/2810817334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d3c0ff750 msgr2=0x7f1d3c19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.178+0000 7f1d41f0e700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d3c0ff750 0x7f1d3c19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d41f0e700 1 -- 172.21.15.115:0/2810817334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d3c0fedc0 msgr2=0x7f1d3c0fe840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d41f0e700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d3c0fedc0 0x7f1d3c0fe840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d41f0e700 1 -- 172.21.15.115:0/2810817334 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d34009d00 con 0x7f1d3c100090 2023-07-14T18:12:12.225 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d40f0c700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d3c0ff750 0x7f1d3c19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:12:12.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d41f0e700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c100090 0x7f1d3c19ccb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1d3400b780 tx=0x7f1d340078e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:12.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d327fc700 1 -- 172.21.15.115:0/2810817334 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3400e040 con 0x7f1d3c100090 2023-07-14T18:12:12.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d327fc700 1 -- 172.21.15.115:0/2810817334 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1d34004c50 con 0x7f1d3c100090 2023-07-14T18:12:12.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c19da30 con 0x7f1d3c100090 2023-07-14T18:12:12.226 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d327fc700 1 -- 172.21.15.115:0/2810817334 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d34005260 con 0x7f1d3c100090 2023-07-14T18:12:12.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.179+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c19de60 con 0x7f1d3c100090 2023-07-14T18:12:12.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.180+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c105b90 con 0x7f1d3c100090 2023-07-14T18:12:12.227 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.181+0000 7f1d327fc700 1 -- 172.21.15.115:0/2810817334 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f1d3401d470 con 0x7f1d3c100090 2023-07-14T18:12:12.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.182+0000 7f1d327fc700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1d280a11e0 0x7f1d280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:12.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.182+0000 7f1d4170d700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1d280a11e0 0x7f1d280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:12.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.182+0000 7f1d327fc700 1 -- 172.21.15.115:0/2810817334 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1d340ce8e0 con 0x7f1d3c100090 2023-07-14T18:12:12.228 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.182+0000 7f1d4170d700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1d280a11e0 0x7f1d280a3690 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c067520 tx=0x7f1d2c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:12.229 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.185+0000 7f1d327fc700 1 -- 172.21.15.115:0/2810817334 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1d34097c50 con 0x7f1d3c100090 2023-07-14T18:12:12.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:12 smithi115 ceph-mon[127451]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-07-14T18:12:12.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:12 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3775197667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:12:12.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:12 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1722676853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:12:12.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:12 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2545342719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:12:12.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:12 smithi150 ceph-mon[128853]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-07-14T18:12:12.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:12 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3775197667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:12:12.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:12 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1722676853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:12:12.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:12 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2545342719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:12:12.414 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.374+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1d3c02ca70 con 0x7f1d3c100090 2023-07-14T18:12:12.415 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.374+0000 7f1d327fc700 1 -- 172.21.15.115:0/2810817334 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1d3401d720 con 0x7f1d3c100090 2023-07-14T18:12:12.415 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 32 2023-07-14T18:12:12.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1d280a11e0 msgr2=0x7f1d280a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1d280a11e0 0x7f1d280a3690 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c067520 tx=0x7f1d2c009260 comp rx=0 tx=0).stop 2023-07-14T18:12:12.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c100090 msgr2=0x7f1d3c19ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.417 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c100090 0x7f1d3c19ccb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1d3400b780 tx=0x7f1d340078e0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 shutdown_connections 2023-07-14T18:12:12.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f1d280a11e0 0x7f1d280a3690 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d3c100090 0x7f1d3c19ccb0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.418 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d3c0fedc0 0x7f1d3c0fe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 --2- 172.21.15.115:0/2810817334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d3c0ff750 0x7f1d3c19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 >> 172.21.15.115:0/2810817334 conn(0x7f1d3c0f98a0 msgr2=0x7f1d3c0fbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:12.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 shutdown_connections 2023-07-14T18:12:12.419 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.376+0000 7f1d43971700 1 -- 172.21.15.115:0/2810817334 wait complete. 2023-07-14T18:12:12.429 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T18:12:12.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:12 smithi187 ceph-mon[129317]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-07-14T18:12:12.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:12 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3775197667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:12:12.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:12 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1722676853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:12:12.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:12 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2545342719' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:12:12.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.511+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/4277862930 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 msgr2=0x7f4ff4106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.511+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/4277862930 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 0x7f4ff4106f50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00b0a0 tx=0x7f4fec00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.552 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.511+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/4277862930 shutdown_connections 2023-07-14T18:12:12.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.511+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/4277862930 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 0x7f4ff4106f50 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.511+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/4277862930 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ff40feb10 0x7f4ff40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.511+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/4277862930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff40ffd00 0x7f4ff4100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.511+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/4277862930 >> 172.21.15.115:0/4277862930 conn(0x7f4ff40fa130 msgr2=0x7f4ff40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:12.553 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.511+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/4277862930 shutdown_connections 2023-07-14T18:12:12.554 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.511+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/4277862930 wait complete. 2023-07-14T18:12:12.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ffb7fb700 1 Processor -- start 2023-07-14T18:12:12.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ffb7fb700 1 -- start start 2023-07-14T18:12:12.556 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ffb7fb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff40feb10 0x7f4ff41958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:12.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ffb7fb700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ff40ffd00 0x7f4ff4195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:12.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ffb7fb700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 0x7f4ff419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:12.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ffb7fb700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff419d470 con 0x7f4ff4100690 2023-07-14T18:12:12.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ffb7fb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff419d5b0 con 0x7f4ff40feb10 2023-07-14T18:12:12.557 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ffb7fb700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff419d740 con 0x7f4ff40ffd00 2023-07-14T18:12:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ff9d98700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 0x7f4ff419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ff9d98700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 0x7f4ff419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54510/0 (socket says 172.21.15.115:54510) 2023-07-14T18:12:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ff9d98700 1 -- 172.21.15.115:0/943075437 learned_addr learned my addr 172.21.15.115:0/943075437 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.512+0000 7f4ff8d96700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ff40ffd00 0x7f4ff4195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ff9597700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff40feb10 0x7f4ff41958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ff9d98700 1 -- 172.21.15.115:0/943075437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ff40ffd00 msgr2=0x7f4ff4195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ff9d98700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ff40ffd00 0x7f4ff4195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ff9d98700 1 -- 172.21.15.115:0/943075437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff40feb10 msgr2=0x7f4ff41958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ff9d98700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff40feb10 0x7f4ff41958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.559 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ff9d98700 1 -- 172.21.15.115:0/943075437 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec009d00 con 0x7f4ff4100690 2023-07-14T18:12:12.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ff8d96700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ff40ffd00 0x7f4ff4195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:12:12.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ff9d98700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 0x7f4ff419cea0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00bad0 tx=0x7f4fec00bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:12.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4fe67fc700 1 -- 172.21.15.115:0/943075437 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec00e040 con 0x7f4ff4100690 2023-07-14T18:12:12.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff419d9c0 con 0x7f4ff4100690 2023-07-14T18:12:12.560 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4fe67fc700 1 -- 172.21.15.115:0/943075437 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4fec004e40 con 0x7f4ff4100690 2023-07-14T18:12:12.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4fe67fc700 1 -- 172.21.15.115:0/943075437 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec0053b0 con 0x7f4ff4100690 2023-07-14T18:12:12.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.513+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff419de90 con 0x7f4ff4100690 2023-07-14T18:12:12.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.514+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff410a1a0 con 0x7f4ff4100690 2023-07-14T18:12:12.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.515+0000 7f4fe67fc700 1 -- 172.21.15.115:0/943075437 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f4fec01da90 con 0x7f4ff4100690 2023-07-14T18:12:12.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.515+0000 7f4fe67fc700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4fe00a11e0 0x7f4fe00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:12.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.516+0000 7f4ff9597700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4fe00a11e0 0x7f4fe00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:12.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.516+0000 7f4fe67fc700 1 -- 172.21.15.115:0/943075437 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4fec0ce980 con 0x7f4ff4100690 2023-07-14T18:12:12.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.516+0000 7f4ff9597700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4fe00a11e0 0x7f4fe00a3690 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40ffb60 tx=0x7f4fe8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:12.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.519+0000 7f4fe67fc700 1 -- 172.21.15.115:0/943075437 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4fec097c90 con 0x7f4ff4100690 2023-07-14T18:12:12.764 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.724+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4ff410a360 con 0x7f4ff4100690 2023-07-14T18:12:12.765 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.724+0000 7f4fe67fc700 1 -- 172.21.15.115:0/943075437 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+600 (secure 0 0 0) 0x7f4fec09b2b0 con 0x7f4ff4100690 2023-07-14T18:12:12.766 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6800/2169355495 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:12.766 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/4031581902 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:12.766 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/3311800921 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:12.767 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6801/2169355495 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:12.767 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2790161712 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:12.767 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2625519889 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:12.768 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6801/568655687 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:12.768 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/3340901626 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:12.769 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:6800/568655687 2023-07-15T18:04:06.647341+0000 2023-07-14T18:12:12.769 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115:0/2416566340 2023-07-15T18:04:28.808488+0000 2023-07-14T18:12:12.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4fe00a11e0 msgr2=0x7f4fe00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4fe00a11e0 0x7f4fe00a3690 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40ffb60 tx=0x7f4fe8009260 comp rx=0 tx=0).stop 2023-07-14T18:12:12.771 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 msgr2=0x7f4ff419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:12.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 0x7f4ff419cea0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00bad0 tx=0x7f4fec00bbb0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 shutdown_connections 2023-07-14T18:12:12.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f4fe00a11e0 0x7f4fe00a3690 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4ff4100690 0x7f4ff419cea0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ff40feb10 0x7f4ff41958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 --2- 172.21.15.115:0/943075437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ff40ffd00 0x7f4ff4195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:12.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 >> 172.21.15.115:0/943075437 conn(0x7f4ff40fa130 msgr2=0x7f4ff40fa900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:12.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 shutdown_connections 2023-07-14T18:12:12.773 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:12.726+0000 7f4ffb7fb700 1 -- 172.21.15.115:0/943075437 wait complete. 2023-07-14T18:12:12.774 INFO:teuthology.orchestra.run.smithi115.stderr:listed 10 entries 2023-07-14T18:12:12.779 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi115.front.sepia.ceph.com... 2023-07-14T18:12:12.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:12.780 DEBUG:teuthology.orchestra.run.smithi115:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-07-14T18:12:12.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi115.stderr:ceph-fuse[177428]: fuse finished with error 0 and tester_r 0 2023-07-14T18:12:12.816 INFO:teuthology.orchestra.run:waiting for 300 2023-07-14T18:12:13.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:13 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1646102224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:12:13.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:13 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2810817334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:12:13.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:13 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/943075437' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:12:13.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:13 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1646102224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:12:13.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:13 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2810817334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:12:13.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:13 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/943075437' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:12:13.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:13 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1646102224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:12:13.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:13 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2810817334' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:12:13.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:13 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/943075437' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:12:14.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:14 smithi115 ceph-mon[127451]: pgmap v311: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2023-07-14T18:12:14.372 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:14 smithi150 ceph-mon[128853]: pgmap v311: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2023-07-14T18:12:14.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:14 smithi187 ceph-mon[129317]: pgmap v311: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2023-07-14T18:12:16.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:16 smithi115 ceph-mon[127451]: pgmap v312: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 6 op/s 2023-07-14T18:12:16.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:16 smithi150 ceph-mon[128853]: pgmap v312: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 6 op/s 2023-07-14T18:12:16.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:16 smithi187 ceph-mon[129317]: pgmap v312: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 853 B/s wr, 6 op/s 2023-07-14T18:12:18.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:18 smithi115 ceph-mon[127451]: pgmap v313: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 7 op/s 2023-07-14T18:12:18.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:18 smithi150 ceph-mon[128853]: pgmap v313: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 7 op/s 2023-07-14T18:12:18.435 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:18 smithi187 ceph-mon[129317]: pgmap v313: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 7 op/s 2023-07-14T18:12:18.918 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi115.front.sepia.ceph.com 2023-07-14T18:12:18.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:18.919 DEBUG:teuthology.orchestra.run.smithi115:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-14T18:12:18.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:18.941 DEBUG:teuthology.orchestra.run.smithi115:> ip netns list 2023-07-14T18:12:19.000 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-14T18:12:19.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:19.001 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T18:12:19.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:12:19.083 DEBUG:teuthology.orchestra.run.smithi115:> sudo ip link delete ceph-brx 2023-07-14T18:12:19.191 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-14T18:12:19.201 INFO:tasks.cephadm:Teardown begin 2023-07-14T18:12:19.202 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-07-14T18:12:19.203 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:12:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.287+0000 7f67a5ce0700 1 -- 172.21.15.115:0/4205510468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00fe280 msgr2=0x7f67a00fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.287+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/4205510468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00fe280 0x7f67a00fe690 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6794009990 tx=0x7f6794009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.287+0000 7f67a5ce0700 1 -- 172.21.15.115:0/4205510468 shutdown_connections 2023-07-14T18:12:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.287+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/4205510468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a00ffe00 0x7f67a01066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.329 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.287+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/4205510468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00fe280 0x7f67a00fe690 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.287+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/4205510468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00ff470 0x7f67a00ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.287+0000 7f67a5ce0700 1 -- 172.21.15.115:0/4205510468 >> 172.21.15.115:0/4205510468 conn(0x7f67a00f98a0 msgr2=0x7f67a00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:19.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.287+0000 7f67a5ce0700 1 -- 172.21.15.115:0/4205510468 shutdown_connections 2023-07-14T18:12:19.330 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.287+0000 7f67a5ce0700 1 -- 172.21.15.115:0/4205510468 wait complete. 2023-07-14T18:12:19.331 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f67a5ce0700 1 Processor -- start 2023-07-14T18:12:19.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f67a5ce0700 1 -- start start 2023-07-14T18:12:19.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f67a5ce0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fe280 0x7f67a0194720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:19.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f67a5ce0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a00ff470 0x7f67a0194c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:19.333 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f67a5ce0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00ffe00 0x7f67a019bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f67a5ce0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a019c2a0 con 0x7f67a00ffe00 2023-07-14T18:12:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f67a5ce0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a019c3e0 con 0x7f67a00fe280 2023-07-14T18:12:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f67a5ce0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a019c570 con 0x7f67a00ff470 2023-07-14T18:12:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f679ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00ffe00 0x7f67a019bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:19.334 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f679ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00ffe00 0x7f67a019bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54524/0 (socket says 172.21.15.115:54524) 2023-07-14T18:12:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f679ffff700 1 -- 172.21.15.115:0/312037298 learned_addr learned my addr 172.21.15.115:0/312037298 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.288+0000 7f679effd700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a00ff470 0x7f67a0194c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679f7fe700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fe280 0x7f67a0194720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679ffff700 1 -- 172.21.15.115:0/312037298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a00ff470 msgr2=0x7f67a0194c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679ffff700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a00ff470 0x7f67a0194c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.335 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679ffff700 1 -- 172.21.15.115:0/312037298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fe280 msgr2=0x7f67a0194720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679ffff700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fe280 0x7f67a0194720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679ffff700 1 -- 172.21.15.115:0/312037298 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67940095f0 con 0x7f67a00ffe00 2023-07-14T18:12:19.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679ffff700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00ffe00 0x7f67a019bcd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6798007ee0 tx=0x7f679800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:19.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679cff9700 1 -- 172.21.15.115:0/312037298 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6798017070 con 0x7f67a00ffe00 2023-07-14T18:12:19.336 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679cff9700 1 -- 172.21.15.115:0/312037298 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f679800f040 con 0x7f67a00ffe00 2023-07-14T18:12:19.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a019c850 con 0x7f67a00ffe00 2023-07-14T18:12:19.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f679cff9700 1 -- 172.21.15.115:0/312037298 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6798005710 con 0x7f67a00ffe00 2023-07-14T18:12:19.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.289+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a019cdb0 con 0x7f67a00ffe00 2023-07-14T18:12:19.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.290+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a0105860 con 0x7f67a00ffe00 2023-07-14T18:12:19.337 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.291+0000 7f679cff9700 1 -- 172.21.15.115:0/312037298 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f6798005870 con 0x7f67a00ffe00 2023-07-14T18:12:19.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.291+0000 7f679cff9700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f67880a12a0 0x7f67880a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:19.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.292+0000 7f679f7fe700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f67880a12a0 0x7f67880a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:19.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.292+0000 7f679cff9700 1 -- 172.21.15.115:0/312037298 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f67980cc530 con 0x7f67a00ffe00 2023-07-14T18:12:19.338 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.292+0000 7f679f7fe700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f67880a12a0 0x7f67880a3750 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f679400c010 tx=0x7f679400b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:19.339 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.296+0000 7f679cff9700 1 -- 172.21.15.115:0/312037298 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6798095810 con 0x7f67a00ffe00 2023-07-14T18:12:19.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.527+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67a002c8e0 con 0x7f67a00ffe00 2023-07-14T18:12:19.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.528+0000 7f679cff9700 1 -- 172.21.15.115:0/312037298 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10851 (secure 0 0 0) 0x7f6798098e30 con 0x7f67a00ffe00 2023-07-14T18:12:19.569 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:12:19.570 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:12:02.979820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24493,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24446":{"gid":24446,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:12:19.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f67880a12a0 msgr2=0x7f67880a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f67880a12a0 0x7f67880a3750 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f679400c010 tx=0x7f679400b540 comp rx=0 tx=0).stop 2023-07-14T18:12:19.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00ffe00 msgr2=0x7f67a019bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00ffe00 0x7f67a019bcd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6798007ee0 tx=0x7f679800d3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 shutdown_connections 2023-07-14T18:12:19.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f67880a12a0 0x7f67880a3750 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f67a00ffe00 0x7f67a019bcd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fe280 0x7f67a0194720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 --2- 172.21.15.115:0/312037298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67a00ff470 0x7f67a0194c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 >> 172.21.15.115:0/312037298 conn(0x7f67a00f98a0 msgr2=0x7f67a01029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:19.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 shutdown_connections 2023-07-14T18:12:19.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.530+0000 7f67a5ce0700 1 -- 172.21.15.115:0/312037298 wait complete. 2023-07-14T18:12:19.577 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 25 2023-07-14T18:12:19.583 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:12:19.706 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.665+0000 7f2d9034c700 1 -- 172.21.15.115:0/4237006283 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880ffcf0 msgr2=0x7f2d88100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.665+0000 7f2d9034c700 1 --2- 172.21.15.115:0/4237006283 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880ffcf0 0x7f2d88100140 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009990 tx=0x7f2d84009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.665+0000 7f2d9034c700 1 -- 172.21.15.115:0/4237006283 shutdown_connections 2023-07-14T18:12:19.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.665+0000 7f2d9034c700 1 --2- 172.21.15.115:0/4237006283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d88100680 0x7f2d88106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.665+0000 7f2d9034c700 1 --2- 172.21.15.115:0/4237006283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d880feb00 0x7f2d880fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.707 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.665+0000 7f2d9034c700 1 --2- 172.21.15.115:0/4237006283 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880ffcf0 0x7f2d88100140 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.665+0000 7f2d9034c700 1 -- 172.21.15.115:0/4237006283 >> 172.21.15.115:0/4237006283 conn(0x7f2d880fa100 msgr2=0x7f2d880fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:19.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.665+0000 7f2d9034c700 1 -- 172.21.15.115:0/4237006283 shutdown_connections 2023-07-14T18:12:19.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.665+0000 7f2d9034c700 1 -- 172.21.15.115:0/4237006283 wait complete. 2023-07-14T18:12:19.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d9034c700 1 Processor -- start 2023-07-14T18:12:19.708 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d9034c700 1 -- start start 2023-07-14T18:12:19.710 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d9034c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880feb00 0x7f2d88068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:19.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d9034c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d880ffcf0 0x7f2d8819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:19.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d9034c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d88100680 0x7f2d8819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:19.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d9034c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8819d3f0 con 0x7f2d880feb00 2023-07-14T18:12:19.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d9034c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8819d560 con 0x7f2d88100680 2023-07-14T18:12:19.711 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d9034c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8819d720 con 0x7f2d880ffcf0 2023-07-14T18:12:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e0e8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880feb00 0x7f2d88068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e0e8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880feb00 0x7f2d88068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54534/0 (socket says 172.21.15.115:54534) 2023-07-14T18:12:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e0e8700 1 -- 172.21.15.115:0/637666487 learned_addr learned my addr 172.21.15.115:0/637666487 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e8e9700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d88100680 0x7f2d8819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:19.712 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8d8e7700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d880ffcf0 0x7f2d8819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:19.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e0e8700 1 -- 172.21.15.115:0/637666487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d880ffcf0 msgr2=0x7f2d8819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e0e8700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d880ffcf0 0x7f2d8819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e0e8700 1 -- 172.21.15.115:0/637666487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d88100680 msgr2=0x7f2d8819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e0e8700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d88100680 0x7f2d8819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.713 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e0e8700 1 -- 172.21.15.115:0/637666487 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d840095f0 con 0x7f2d880feb00 2023-07-14T18:12:19.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8d8e7700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d880ffcf0 0x7f2d8819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:12:19.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.666+0000 7f2d8e0e8700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880feb00 0x7f2d88068330 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00c970 tx=0x7f2d7c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:19.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.667+0000 7f2d7b7fe700 1 -- 172.21.15.115:0/637666487 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c004220 con 0x7f2d880feb00 2023-07-14T18:12:19.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.667+0000 7f2d7b7fe700 1 -- 172.21.15.115:0/637666487 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2d7c007d30 con 0x7f2d880feb00 2023-07-14T18:12:19.714 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.667+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8819da00 con 0x7f2d880feb00 2023-07-14T18:12:19.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.667+0000 7f2d7b7fe700 1 -- 172.21.15.115:0/637666487 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c003b30 con 0x7f2d880feb00 2023-07-14T18:12:19.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.667+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8819df10 con 0x7f2d880feb00 2023-07-14T18:12:19.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.668+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d88106140 con 0x7f2d880feb00 2023-07-14T18:12:19.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.669+0000 7f2d7b7fe700 1 -- 172.21.15.115:0/637666487 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f2d7c003c90 con 0x7f2d880feb00 2023-07-14T18:12:19.715 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.669+0000 7f2d7b7fe700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2d740a1220 0x7f2d740a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:19.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.669+0000 7f2d8d8e7700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2d740a1220 0x7f2d740a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:19.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.669+0000 7f2d7b7fe700 1 -- 172.21.15.115:0/637666487 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2d7c0cc9d0 con 0x7f2d880feb00 2023-07-14T18:12:19.716 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.670+0000 7f2d8d8e7700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2d740a1220 0x7f2d740a36d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2d84000c00 tx=0x7f2d84005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:19.717 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.673+0000 7f2d7b7fe700 1 -- 172.21.15.115:0/637666487 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2d7c095cb0 con 0x7f2d880feb00 2023-07-14T18:12:19.947 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.907+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d8802c8e0 con 0x7f2d880feb00 2023-07-14T18:12:19.948 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.907+0000 7f2d7b7fe700 1 -- 172.21.15.115:0/637666487 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10851 (secure 0 0 0) 0x7f2d7c0992d0 con 0x7f2d880feb00 2023-07-14T18:12:19.949 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:12:19.949 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/3105890615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":3105890615},{"type":"v1","addr":"172.21.15.115:6835","nonce":3105890615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/750049188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":750049188},{"type":"v1","addr":"172.21.15.115:6837","nonce":750049188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/2547274817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":2547274817},{"type":"v1","addr":"172.21.15.115:6839","nonce":2547274817}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6841/2429034282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6840","nonce":2429034282},{"type":"v1","addr":"172.21.15.115:6841","nonce":2429034282}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1440309649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1440309649},{"type":"v1","addr":"172.21.15.150:6833","nonce":1440309649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6833/1471546028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":1471546028},{"type":"v1","addr":"172.21.15.187:6833","nonce":1471546028}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:10:17.634244+0000","modified":"2023-07-14T18:12:02.979820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24515,"mds_1":24493,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24446":{"gid":24446,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/3630781789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3630781789},{"type":"v1","addr":"172.21.15.150:6835","nonce":3630781789}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.187:6835/4258537735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6834","nonce":4258537735},{"type":"v1","addr":"172.21.15.187:6835","nonce":4258537735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24475":{"gid":24475,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.187:6837/771796341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6836","nonce":771796341},{"type":"v1","addr":"172.21.15.187:6837","nonce":771796341}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6837/1462100932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1462100932},{"type":"v1","addr":"172.21.15.150:6837","nonce":1462100932}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.187:6839/1449243194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6838","nonce":1449243194},{"type":"v1","addr":"172.21.15.187:6839","nonce":1449243194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6839/2895627668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":2895627668},{"type":"v1","addr":"172.21.15.150:6839","nonce":2895627668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:12:19.950 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2d740a1220 msgr2=0x7f2d740a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2d740a1220 0x7f2d740a36d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2d84000c00 tx=0x7f2d84005fb0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.951 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880feb00 msgr2=0x7f2d88068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:19.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880feb00 0x7f2d88068330 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00c970 tx=0x7f2d7c00cc80 comp rx=0 tx=0).stop 2023-07-14T18:12:19.952 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 shutdown_connections 2023-07-14T18:12:19.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f2d740a1220 0x7f2d740a36d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d88100680 0x7f2d8819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.953 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d880feb00 0x7f2d88068330 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 --2- 172.21.15.115:0/637666487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d880ffcf0 0x7f2d8819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:19.954 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 >> 172.21.15.115:0/637666487 conn(0x7f2d880fa100 msgr2=0x7f2d88103280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:19.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 shutdown_connections 2023-07-14T18:12:19.955 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:19.909+0000 7f2d9034c700 1 -- 172.21.15.115:0/637666487 wait complete. 2023-07-14T18:12:19.956 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 25 2023-07-14T18:12:19.963 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:12:20.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 -- 172.21.15.115:0/1069414816 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c103020 msgr2=0x7f620c105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 --2- 172.21.15.115:0/1069414816 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c103020 0x7f620c105400 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009990 tx=0x7f61fc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.087 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 -- 172.21.15.115:0/1069414816 shutdown_connections 2023-07-14T18:12:20.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 --2- 172.21.15.115:0/1069414816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f620c105940 0x7f620c107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 --2- 172.21.15.115:0/1069414816 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c100700 0x7f620c102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 --2- 172.21.15.115:0/1069414816 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c103020 0x7f620c105400 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.088 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 -- 172.21.15.115:0/1069414816 >> 172.21.15.115:0/1069414816 conn(0x7f620c0fa100 msgr2=0x7f620c0fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:20.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 -- 172.21.15.115:0/1069414816 shutdown_connections 2023-07-14T18:12:20.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 -- 172.21.15.115:0/1069414816 wait complete. 2023-07-14T18:12:20.089 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.046+0000 7f6212208700 1 Processor -- start 2023-07-14T18:12:20.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f6212208700 1 -- start start 2023-07-14T18:12:20.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f6212208700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f620c100700 0x7f620c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.091 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f6212208700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c103020 0x7f620c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f6212208700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c105940 0x7f620c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f6212208700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c19f560 con 0x7f620c105940 2023-07-14T18:12:20.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f6212208700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c19f6a0 con 0x7f620c100700 2023-07-14T18:12:20.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f6212208700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c19f830 con 0x7f620c103020 2023-07-14T18:12:20.092 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c105940 0x7f620c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c105940 0x7f620c19ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54552/0 (socket says 172.21.15.115:54552) 2023-07-14T18:12:20.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620bfff700 1 -- 172.21.15.115:0/723715323 learned_addr learned my addr 172.21.15.115:0/723715323 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:20.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620b7fe700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f620c100700 0x7f620c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.093 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620affd700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c103020 0x7f620c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620bfff700 1 -- 172.21.15.115:0/723715323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c103020 msgr2=0x7f620c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620bfff700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c103020 0x7f620c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620bfff700 1 -- 172.21.15.115:0/723715323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f620c100700 msgr2=0x7f620c1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620bfff700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f620c100700 0x7f620c1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.094 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620bfff700 1 -- 172.21.15.115:0/723715323 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc0095f0 con 0x7f620c105940 2023-07-14T18:12:20.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620b7fe700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f620c100700 0x7f620c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:12:20.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f620bfff700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c105940 0x7f620c19ef90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6204007d70 tx=0x7f620400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:20.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.047+0000 7f6208ff9700 1 -- 172.21.15.115:0/723715323 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620400ad40 con 0x7f620c105940 2023-07-14T18:12:20.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.048+0000 7f6208ff9700 1 -- 172.21.15.115:0/723715323 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f62040128b0 con 0x7f620c105940 2023-07-14T18:12:20.095 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.048+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c19fb10 con 0x7f620c105940 2023-07-14T18:12:20.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.048+0000 7f6208ff9700 1 -- 172.21.15.115:0/723715323 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62040116b0 con 0x7f620c105940 2023-07-14T18:12:20.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.048+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c19ffd0 con 0x7f620c105940 2023-07-14T18:12:20.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.048+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c1074d0 con 0x7f620c105940 2023-07-14T18:12:20.096 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.050+0000 7f6208ff9700 1 -- 172.21.15.115:0/723715323 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f6204012a20 con 0x7f620c105940 2023-07-14T18:12:20.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.050+0000 7f6208ff9700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f61f80a1220 0x7f61f80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.050+0000 7f6208ff9700 1 -- 172.21.15.115:0/723715323 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f62040cd3b0 con 0x7f620c105940 2023-07-14T18:12:20.097 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.050+0000 7f620b7fe700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f61f80a1220 0x7f61f80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.051+0000 7f620b7fe700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f61f80a1220 0x7f61f80a36d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f61f4005fd0 tx=0x7f61f4005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:20.098 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.054+0000 7f6208ff9700 1 -- 172.21.15.115:0/723715323 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f62040967a0 con 0x7f620c105940 2023-07-14T18:12:20.284 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.243+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f620c02caf0 con 0x7f620c105940 2023-07-14T18:12:20.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.245+0000 7f6208ff9700 1 -- 172.21.15.115:0/723715323 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24558 (secure 0 0 0) 0x7f6204099dc0 con 0x7f620c105940 2023-07-14T18:12:20.286 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-07-14T18:12:20.287 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":89,"fsid":"a39f385c-2270-11ee-9b33-001a4aab830c","created":"2023-07-14T18:03:38.101205+0000","modified":"2023-07-14T18:11:30.172466+0000","last_up_change":"2023-07-14T18:09:36.617979+0000","last_in_change":"2023-07-14T18:09:18.489699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:06:31.134937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:10:15.471213+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:10:16.791973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:10:20.663275+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"578188a6-96fa-444a-8a11-224bb5a7c098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6803","nonce":2677759926}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6805","nonce":2677759926}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6809","nonce":2677759926}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2677759926},{"type":"v1","addr":"172.21.15.115:6807","nonce":2677759926}]},"public_addr":"172.21.15.115:6803/2677759926","cluster_addr":"172.21.15.115:6805/2677759926","heartbeat_back_addr":"172.21.15.115:6809/2677759926","heartbeat_front_addr":"172.21.15.115:6807/2677759926","state":["exists","up"]},{"osd":1,"uuid":"ffda5af6-01b2-4a92-863f-7f6518223778","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6801","nonce":396669416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6803","nonce":396669416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6807","nonce":396669416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":396669416},{"type":"v1","addr":"172.21.15.150:6805","nonce":396669416}]},"public_addr":"172.21.15.150:6801/396669416","cluster_addr":"172.21.15.150:6803/396669416","heartbeat_back_addr":"172.21.15.150:6807/396669416","heartbeat_front_addr":"172.21.15.150:6805/396669416","state":["exists","up"]},{"osd":2,"uuid":"83a08d79-e0ac-47ea-a061-5b92c4a33d3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6801","nonce":262076123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6803","nonce":262076123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6807","nonce":262076123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":262076123},{"type":"v1","addr":"172.21.15.187:6805","nonce":262076123}]},"public_addr":"172.21.15.187:6801/262076123","cluster_addr":"172.21.15.187:6803/262076123","heartbeat_back_addr":"172.21.15.187:6807/262076123","heartbeat_front_addr":"172.21.15.187:6805/262076123","state":["exists","up"]},{"osd":3,"uuid":"9796f71f-7cff-4ee0-ae32-545fb5fc886d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6811","nonce":3432098119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6813","nonce":3432098119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6817","nonce":3432098119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":3432098119},{"type":"v1","addr":"172.21.15.115:6815","nonce":3432098119}]},"public_addr":"172.21.15.115:6811/3432098119","cluster_addr":"172.21.15.115:6813/3432098119","heartbeat_back_addr":"172.21.15.115:6817/3432098119","heartbeat_front_addr":"172.21.15.115:6815/3432098119","state":["exists","up"]},{"osd":4,"uuid":"82386140-1f12-43e9-9d70-6ca597009e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6809","nonce":3033896975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6811","nonce":3033896975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6815","nonce":3033896975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3033896975},{"type":"v1","addr":"172.21.15.150:6813","nonce":3033896975}]},"public_addr":"172.21.15.150:6809/3033896975","cluster_addr":"172.21.15.150:6811/3033896975","heartbeat_back_addr":"172.21.15.150:6815/3033896975","heartbeat_front_addr":"172.21.15.150:6813/3033896975","state":["exists","up"]},{"osd":5,"uuid":"9af1e805-b05f-422e-8f27-f95d74e9fec6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6809","nonce":3577781836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6811","nonce":3577781836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6815","nonce":3577781836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3577781836},{"type":"v1","addr":"172.21.15.187:6813","nonce":3577781836}]},"public_addr":"172.21.15.187:6809/3577781836","cluster_addr":"172.21.15.187:6811/3577781836","heartbeat_back_addr":"172.21.15.187:6815/3577781836","heartbeat_front_addr":"172.21.15.187:6813/3577781836","state":["exists","up"]},{"osd":6,"uuid":"5bceb601-e8f0-440f-b674-3ce286cc4d3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6819","nonce":1235073533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6821","nonce":1235073533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6825","nonce":1235073533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":1235073533},{"type":"v1","addr":"172.21.15.115:6823","nonce":1235073533}]},"public_addr":"172.21.15.115:6819/1235073533","cluster_addr":"172.21.15.115:6821/1235073533","heartbeat_back_addr":"172.21.15.115:6825/1235073533","heartbeat_front_addr":"172.21.15.115:6823/1235073533","state":["exists","up"]},{"osd":7,"uuid":"bbe475b0-b4f9-41df-9615-a90c6fc62280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6817","nonce":3128310736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6819","nonce":3128310736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6823","nonce":3128310736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3128310736},{"type":"v1","addr":"172.21.15.150:6821","nonce":3128310736}]},"public_addr":"172.21.15.150:6817/3128310736","cluster_addr":"172.21.15.150:6819/3128310736","heartbeat_back_addr":"172.21.15.150:6823/3128310736","heartbeat_front_addr":"172.21.15.150:6821/3128310736","state":["exists","up"]},{"osd":8,"uuid":"574a1fb8-fa71-4c03-812d-bb1d8da70f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6817","nonce":1513660091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6819","nonce":1513660091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6823","nonce":1513660091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1513660091},{"type":"v1","addr":"172.21.15.187:6821","nonce":1513660091}]},"public_addr":"172.21.15.187:6817/1513660091","cluster_addr":"172.21.15.187:6819/1513660091","heartbeat_back_addr":"172.21.15.187:6823/1513660091","heartbeat_front_addr":"172.21.15.187:6821/1513660091","state":["exists","up"]},{"osd":9,"uuid":"fd79ae74-eaa1-4b63-aad8-c77864ba2eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6827","nonce":3384303720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6829","nonce":3384303720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6833","nonce":3384303720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3384303720},{"type":"v1","addr":"172.21.15.115:6831","nonce":3384303720}]},"public_addr":"172.21.15.115:6827/3384303720","cluster_addr":"172.21.15.115:6829/3384303720","heartbeat_back_addr":"172.21.15.115:6833/3384303720","heartbeat_front_addr":"172.21.15.115:6831/3384303720","state":["exists","up"]},{"osd":10,"uuid":"2d66bdb3-b20d-4474-ab4b-9d0f81232272","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6825","nonce":4251241728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6827","nonce":4251241728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6831","nonce":4251241728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4251241728},{"type":"v1","addr":"172.21.15.150:6829","nonce":4251241728}]},"public_addr":"172.21.15.150:6825/4251241728","cluster_addr":"172.21.15.150:6827/4251241728","heartbeat_back_addr":"172.21.15.150:6831/4251241728","heartbeat_front_addr":"172.21.15.150:6829/4251241728","state":["exists","up"]},{"osd":11,"uuid":"646460b3-b30a-4570-8b1a-dce626e03f2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6825","nonce":2714221245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6827","nonce":2714221245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6831","nonce":2714221245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":2714221245},{"type":"v1","addr":"172.21.15.187:6829","nonce":2714221245}]},"public_addr":"172.21.15.187:6825/2714221245","cluster_addr":"172.21.15.187:6827/2714221245","heartbeat_back_addr":"172.21.15.187:6831/2714221245","heartbeat_front_addr":"172.21.15.187:6829/2714221245","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:05:48.746152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:08.998920+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:29.422306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:06:50.174210+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:09.780327+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:30.239344+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:07:51.239694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:11.488254+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:32.072859+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:08:53.208373+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:13.419719+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:09:34.731021+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6800/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/4031581902":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/3311800921":"2023-07-15T18:04:28.808488+0000","172.21.15.115:6801/2169355495":"2023-07-15T18:04:28.808488+0000","172.21.15.115:0/2790161712":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2625519889":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6801/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/3340901626":"2023-07-15T18:04:06.647341+0000","172.21.15.115:6800/568655687":"2023-07-15T18:04:06.647341+0000","172.21.15.115:0/2416566340":"2023-07-15T18:04:28.808488+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:12:20.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f61f80a1220 msgr2=0x7f61f80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f61f80a1220 0x7f61f80a36d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f61f4005fd0 tx=0x7f61f4005ee0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.288 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c105940 msgr2=0x7f620c19ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c105940 0x7f620c19ef90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6204007d70 tx=0x7f620400a340 comp rx=0 tx=0).stop 2023-07-14T18:12:20.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 shutdown_connections 2023-07-14T18:12:20.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f61f80a1220 0x7f61f80a36d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f620c105940 0x7f620c19ef90 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f620c100700 0x7f620c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 --2- 172.21.15.115:0/723715323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f620c103020 0x7f620c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 >> 172.21.15.115:0/723715323 conn(0x7f620c0fa100 msgr2=0x7f620c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:20.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 shutdown_connections 2023-07-14T18:12:20.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.247+0000 7f6212208700 1 -- 172.21.15.115:0/723715323 wait complete. 2023-07-14T18:12:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:20 smithi115 ceph-mon[127451]: pgmap v314: 129 pgs: 129 active+clean; 646 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 7 op/s 2023-07-14T18:12:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:20 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/312037298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:12:20.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:20 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/637666487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:12:20.302 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-07-14T18:12:20.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-mon[128853]: pgmap v314: 129 pgs: 129 active+clean; 646 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 7 op/s 2023-07-14T18:12:20.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/312037298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:12:20.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/637666487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:12:20.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.384+0000 7fd39b9a6700 1 -- 172.21.15.115:0/221547117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3941006c0 msgr2=0x7fd394106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.384+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/221547117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3941006c0 0x7fd394106f80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd38c00b0a0 tx=0x7fd38c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.384+0000 7fd39b9a6700 1 -- 172.21.15.115:0/221547117 shutdown_connections 2023-07-14T18:12:20.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.384+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/221547117 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3941006c0 0x7fd394106f80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.384+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/221547117 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3940feb40 0x7fd3940fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.384+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/221547117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3940ffd30 0x7fd394100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.384+0000 7fd39b9a6700 1 -- 172.21.15.115:0/221547117 >> 172.21.15.115:0/221547117 conn(0x7fd3940fa160 msgr2=0x7fd3940fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:20.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.384+0000 7fd39b9a6700 1 -- 172.21.15.115:0/221547117 shutdown_connections 2023-07-14T18:12:20.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.384+0000 7fd39b9a6700 1 -- 172.21.15.115:0/221547117 wait complete. 2023-07-14T18:12:20.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd39b9a6700 1 Processor -- start 2023-07-14T18:12:20.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd39b9a6700 1 -- start start 2023-07-14T18:12:20.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd39b9a6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3940feb40 0x7fd3941948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd39b9a6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3940ffd30 0x7fd394194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd39b9a6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3941006c0 0x7fd39419be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd39b9a6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39419c420 con 0x7fd3940ffd30 2023-07-14T18:12:20.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd39b9a6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39419c560 con 0x7fd3941006c0 2023-07-14T18:12:20.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd39b9a6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39419c6f0 con 0x7fd3940feb40 2023-07-14T18:12:20.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd398f41700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3940ffd30 0x7fd394194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd398f41700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3940ffd30 0x7fd394194de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54564/0 (socket says 172.21.15.115:54564) 2023-07-14T18:12:20.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd398f41700 1 -- 172.21.15.115:0/2456058187 learned_addr learned my addr 172.21.15.115:0/2456058187 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:20.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd399742700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3940feb40 0x7fd3941948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd399f43700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3941006c0 0x7fd39419be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd398f41700 1 -- 172.21.15.115:0/2456058187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3940feb40 msgr2=0x7fd3941948a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd398f41700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3940feb40 0x7fd3941948a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd398f41700 1 -- 172.21.15.115:0/2456058187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3941006c0 msgr2=0x7fd39419be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd398f41700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3941006c0 0x7fd39419be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.385+0000 7fd398f41700 1 -- 172.21.15.115:0/2456058187 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c009d00 con 0x7fd3940ffd30 2023-07-14T18:12:20.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.386+0000 7fd398f41700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3940ffd30 0x7fd394194de0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd39000d950 tx=0x7fd39000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:20.433 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.386+0000 7fd3867fc700 1 -- 172.21.15.115:0/2456058187 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390009940 con 0x7fd3940ffd30 2023-07-14T18:12:20.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.386+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39419c9d0 con 0x7fd3940ffd30 2023-07-14T18:12:20.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.386+0000 7fd3867fc700 1 -- 172.21.15.115:0/2456058187 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd390010460 con 0x7fd3940ffd30 2023-07-14T18:12:20.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.386+0000 7fd3867fc700 1 -- 172.21.15.115:0/2456058187 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39000f620 con 0x7fd3940ffd30 2023-07-14T18:12:20.434 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.386+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39419cf30 con 0x7fd3940ffd30 2023-07-14T18:12:20.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.387+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3941060f0 con 0x7fd3940ffd30 2023-07-14T18:12:20.435 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.388+0000 7fd3867fc700 1 -- 172.21.15.115:0/2456058187 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fd3900105d0 con 0x7fd3940ffd30 2023-07-14T18:12:20.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.388+0000 7fd3867fc700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3800a11e0 0x7fd3800a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.389+0000 7fd399742700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3800a11e0 0x7fd3800a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.436 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.389+0000 7fd3867fc700 1 -- 172.21.15.115:0/2456058187 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd3900cccc0 con 0x7fd3940ffd30 2023-07-14T18:12:20.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.389+0000 7fd399742700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3800a11e0 0x7fd3800a3690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd3940ffb90 tx=0x7fd38800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:20.437 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.392+0000 7fd3867fc700 1 -- 172.21.15.115:0/2456058187 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd390095fa0 con 0x7fd3940ffd30 2023-07-14T18:12:20.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-mon[129317]: pgmap v314: 129 pgs: 129 active+clean; 646 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 7 op/s 2023-07-14T18:12:20.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/312037298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:12:20.438 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/637666487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:12:20.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.625+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd39402c8e0 con 0x7fd3940ffd30 2023-07-14T18:12:20.672 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.631+0000 7fd3867fc700 1 -- 172.21.15.115:0/2456058187 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd3900995c0 con 0x7fd3940ffd30 2023-07-14T18:12:20.674 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3800a11e0 msgr2=0x7fd3800a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3800a11e0 0x7fd3800a3690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd3940ffb90 tx=0x7fd38800b410 comp rx=0 tx=0).stop 2023-07-14T18:12:20.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3940ffd30 msgr2=0x7fd394194de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3940ffd30 0x7fd394194de0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd39000d950 tx=0x7fd39000dc60 comp rx=0 tx=0).stop 2023-07-14T18:12:20.675 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 shutdown_connections 2023-07-14T18:12:20.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fd3800a11e0 0x7fd3800a3690 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3941006c0 0x7fd39419be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd3940feb40 0x7fd3941948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 --2- 172.21.15.115:0/2456058187 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd3940ffd30 0x7fd394194de0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 >> 172.21.15.115:0/2456058187 conn(0x7fd3940fa160 msgr2=0x7fd394103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:20.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 shutdown_connections 2023-07-14T18:12:20.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.633+0000 7fd39b9a6700 1 -- 172.21.15.115:0/2456058187 wait complete. 2023-07-14T18:12:20.679 INFO:teuthology.orchestra.run.smithi115.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-07-14T18:12:20.689 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-07-14T18:12:20.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.773+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/3570146198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 msgr2=0x7fc5c80fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.773+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/3570146198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 0x7fc5c80fef50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009990 tx=0x7fc5b8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.773+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/3570146198 shutdown_connections 2023-07-14T18:12:20.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.773+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/3570146198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c81006c0 0x7fc5c8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.815 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.773+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/3570146198 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 0x7fc5c80fef50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.773+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/3570146198 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5c80ffd30 0x7fc5c8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.773+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/3570146198 >> 172.21.15.115:0/3570146198 conn(0x7fc5c80fa160 msgr2=0x7fc5c80fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:20.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.773+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/3570146198 shutdown_connections 2023-07-14T18:12:20.816 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.773+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/3570146198 wait complete. 2023-07-14T18:12:20.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5ce8a3700 1 Processor -- start 2023-07-14T18:12:20.817 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5ce8a3700 1 -- start start 2023-07-14T18:12:20.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5ce8a3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 0x7fc5c8069540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.819 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5ce8a3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c80ffd30 0x7fc5c819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5ce8a3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5c81006c0 0x7fc5c819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5ce8a3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c819d3f0 con 0x7fc5c80feb40 2023-07-14T18:12:20.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5ce8a3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c819d560 con 0x7fc5c80ffd30 2023-07-14T18:12:20.820 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5ce8a3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c819d720 con 0x7fc5c81006c0 2023-07-14T18:12:20.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5c7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 0x7fc5c8069540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5c7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 0x7fc5c8069540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54576/0 (socket says 172.21.15.115:54576) 2023-07-14T18:12:20.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5c7fff700 1 -- 172.21.15.115:0/1444853747 learned_addr learned my addr 172.21.15.115:0/1444853747 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:20.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5cce40700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5c81006c0 0x7fc5c819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5c77fe700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c80ffd30 0x7fc5c819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5c7fff700 1 -- 172.21.15.115:0/1444853747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5c81006c0 msgr2=0x7fc5c819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5c7fff700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5c81006c0 0x7fc5c819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5c7fff700 1 -- 172.21.15.115:0/1444853747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c80ffd30 msgr2=0x7fc5c819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:20.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5c7fff700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c80ffd30 0x7fc5c819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:20.822 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.774+0000 7fc5c7fff700 1 -- 172.21.15.115:0/1444853747 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80095f0 con 0x7fc5c80feb40 2023-07-14T18:12:20.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.775+0000 7fc5c7fff700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 0x7fc5c8069540 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8000c00 tx=0x7fc5b80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:20.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.775+0000 7fc5c57fa700 1 -- 172.21.15.115:0/1444853747 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b801d070 con 0x7fc5c80feb40 2023-07-14T18:12:20.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.775+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c819d9a0 con 0x7fc5c80feb40 2023-07-14T18:12:20.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.775+0000 7fc5c57fa700 1 -- 172.21.15.115:0/1444853747 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc5b8003f40 con 0x7fc5c80feb40 2023-07-14T18:12:20.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.775+0000 7fc5c57fa700 1 -- 172.21.15.115:0/1444853747 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8022bf0 con 0x7fc5c80feb40 2023-07-14T18:12:20.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.775+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c819de50 con 0x7fc5c80feb40 2023-07-14T18:12:20.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.776+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c8105b80 con 0x7fc5c80feb40 2023-07-14T18:12:20.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.777+0000 7fc5c57fa700 1 -- 172.21.15.115:0/1444853747 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fc5b8022d50 con 0x7fc5c80feb40 2023-07-14T18:12:20.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.777+0000 7fc5c57fa700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5b00a11e0 0x7fc5b00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:20.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.778+0000 7fc5c77fe700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5b00a11e0 0x7fc5b00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:20.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.778+0000 7fc5c57fa700 1 -- 172.21.15.115:0/1444853747 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fc5b80ce060 con 0x7fc5c80feb40 2023-07-14T18:12:20.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.778+0000 7fc5c77fe700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5b00a11e0 0x7fc5b00a3690 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80692b0 tx=0x7fc5bc0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:20.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:20.782+0000 7fc5c57fa700 1 -- 172.21.15.115:0/1444853747 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc5b80972d0 con 0x7fc5c80feb40 2023-07-14T18:12:21.055 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.014+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc5c802cb00 con 0x7fc5c80feb40 2023-07-14T18:12:21.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:20 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-a[127427]: 2023-07-14T18:12:20.626+0000 7f29832c8700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T18:12:21.075 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-c[155141]: ignoring --setuser ceph since I am not root 2023-07-14T18:12:21.075 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-c[155141]: ignoring --setgroup ceph since I am not root 2023-07-14T18:12:21.075 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-mds[155165]: -- 172.21.15.187:0/771647024 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f2dfe2f1e0 con 0x55f2dfe05000 2023-07-14T18:12:21.075 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-c[155141]: starting mds.c at 2023-07-14T18:12:21.076 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-i[155482]: ignoring --setuser ceph since I am not root 2023-07-14T18:12:21.076 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-i[155482]: ignoring --setgroup ceph since I am not root 2023-07-14T18:12:21.076 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-mds[155506]: -- 172.21.15.187:0/3148368749 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5596d89071e0 con 0x5596d88dc800 2023-07-14T18:12:21.077 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-i[155482]: starting mds.i at 2023-07-14T18:12:21.077 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-f[154799]: ignoring --setuser ceph since I am not root 2023-07-14T18:12:21.077 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-f[154799]: ignoring --setgroup ceph since I am not root 2023-07-14T18:12:21.077 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-mds[154823]: -- 172.21.15.187:0/2199826504 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555e3b9011e0 con 0x555e3b8d6800 2023-07-14T18:12:21.077 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:12:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-f[154799]: starting mds.f at 2023-07-14T18:12:21.078 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-b[154137]: ignoring --setuser ceph since I am not root 2023-07-14T18:12:21.078 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-b[154137]: ignoring --setgroup ceph since I am not root 2023-07-14T18:12:21.078 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-mds[154161]: -- 172.21.15.150:0/218251781 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5651fb8891e0 con 0x5651fb85e800 2023-07-14T18:12:21.079 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-b[154137]: starting mds.b at 2023-07-14T18:12:21.079 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-h[153682]: ignoring --setuser ceph since I am not root 2023-07-14T18:12:21.079 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-h[153682]: ignoring --setgroup ceph since I am not root 2023-07-14T18:12:21.080 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-mds[153707]: -- 172.21.15.150:0/51615391 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563c7ef411e0 con 0x563c7ef16c00 2023-07-14T18:12:21.080 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-h[153682]: starting mds.h at 2023-07-14T18:12:21.080 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-e[153341]: ignoring --setuser ceph since I am not root 2023-07-14T18:12:21.081 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-e[153341]: ignoring --setgroup ceph since I am not root 2023-07-14T18:12:21.081 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-mds[153365]: -- 172.21.15.150:0/37679063 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562d4154b1e0 con 0x562d41520c00 2023-07-14T18:12:21.081 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:12:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-e[153341]: starting mds.e at 2023-07-14T18:12:21.357 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:21 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/723715323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:12:21.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:21 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2456058187' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T18:12:21.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:21 smithi115 ceph-mon[127451]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T18:12:21.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:21 smithi115 ceph-mon[127451]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T18:12:21.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:21 smithi115 ceph-mon[127451]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T18:12:21.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:21 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/2456058187' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T18:12:21.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:21 smithi115 ceph-mon[127451]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T18:12:21.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:21 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1444853747' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:12:21.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:21 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/723715323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:12:21.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:21 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2456058187' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T18:12:21.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:21 smithi150 ceph-mon[128853]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T18:12:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:21 smithi150 ceph-mon[128853]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T18:12:21.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:21 smithi150 ceph-mon[128853]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T18:12:21.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:21 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/2456058187' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T18:12:21.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:21 smithi150 ceph-mon[128853]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T18:12:21.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:21 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1444853747' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:12:21.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:21 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/723715323' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:12:21.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:21 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2456058187' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T18:12:21.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:21 smithi187 ceph-mon[129317]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T18:12:21.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:21 smithi187 ceph-mon[129317]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T18:12:21.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:21 smithi187 ceph-mon[129317]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T18:12:21.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:21 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/2456058187' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T18:12:21.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:21 smithi187 ceph-mon[129317]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T18:12:21.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:21 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1444853747' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:12:21.676 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.635+0000 7fc5c57fa700 1 -- 172.21.15.115:0/1444853747 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc5b809a8f0 con 0x7fc5c80feb40 2023-07-14T18:12:21.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5b00a11e0 msgr2=0x7fc5b00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:21.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5b00a11e0 0x7fc5b00a3690 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80692b0 tx=0x7fc5bc0073d0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.680 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 msgr2=0x7fc5c8069540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:21.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 0x7fc5c8069540 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8000c00 tx=0x7fc5b80047b0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 shutdown_connections 2023-07-14T18:12:21.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fc5b00a11e0 0x7fc5b00a3690 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5c81006c0 0x7fc5c819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5c80feb40 0x7fc5c8069540 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 --2- 172.21.15.115:0/1444853747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c80ffd30 0x7fc5c819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 >> 172.21.15.115:0/1444853747 conn(0x7fc5c80fa160 msgr2=0x7fc5c80fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:21.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 shutdown_connections 2023-07-14T18:12:21.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.638+0000 7fc5ce8a3700 1 -- 172.21.15.115:0/1444853747 wait complete. 2023-07-14T18:12:21.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-07-14T18:12:21.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.781+0000 7f794e6ee700 1 -- 172.21.15.115:0/897477645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79480ff750 msgr2=0x7f7948107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:21.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.781+0000 7f794e6ee700 1 --2- 172.21.15.115:0/897477645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79480ff750 0x7f7948107020 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f794000b0a0 tx=0x7f794000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.781+0000 7f794e6ee700 1 -- 172.21.15.115:0/897477645 shutdown_connections 2023-07-14T18:12:21.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.781+0000 7f794e6ee700 1 --2- 172.21.15.115:0/897477645 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f79480ff750 0x7f7948107020 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.781+0000 7f794e6ee700 1 --2- 172.21.15.115:0/897477645 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7948100090 0x7f79481004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.781+0000 7f794e6ee700 1 --2- 172.21.15.115:0/897477645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79480fedc0 0x7f79480ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.781+0000 7f794e6ee700 1 -- 172.21.15.115:0/897477645 >> 172.21.15.115:0/897477645 conn(0x7f79480f98a0 msgr2=0x7f79480fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:21.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 -- 172.21.15.115:0/897477645 shutdown_connections 2023-07-14T18:12:21.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 -- 172.21.15.115:0/897477645 wait complete. 2023-07-14T18:12:21.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 Processor -- start 2023-07-14T18:12:21.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 -- start start 2023-07-14T18:12:21.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79480fedc0 0x7f79481948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:21.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480ff750 0x7f7948194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:21.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7948100090 0x7f794819be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:21.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794819c420 con 0x7f7948100090 2023-07-14T18:12:21.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794819c560 con 0x7f79480fedc0 2023-07-14T18:12:21.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794e6ee700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794819c6f0 con 0x7f79480ff750 2023-07-14T18:12:21.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.782+0000 7f794cc8b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7948100090 0x7f794819be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:21.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794cc8b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7948100090 0x7f794819be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54594/0 (socket says 172.21.15.115:54594) 2023-07-14T18:12:21.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794cc8b700 1 -- 172.21.15.115:0/3295507248 learned_addr learned my addr 172.21.15.115:0/3295507248 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:21.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f7947fff700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79480fedc0 0x7f79481948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:21.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794cc8b700 1 -- 172.21.15.115:0/3295507248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480ff750 msgr2=0x7f7948194de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:21.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f79477fe700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480ff750 0x7f7948194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:21.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794cc8b700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480ff750 0x7f7948194de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794cc8b700 1 -- 172.21.15.115:0/3295507248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79480fedc0 msgr2=0x7f79481948a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:21.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794cc8b700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79480fedc0 0x7f79481948a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:21.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794cc8b700 1 -- 172.21.15.115:0/3295507248 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940009d00 con 0x7f7948100090 2023-07-14T18:12:21.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f7947fff700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79480fedc0 0x7f79481948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:12:21.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f79477fe700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480ff750 0x7f7948194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:12:21.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794cc8b700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7948100090 0x7f794819be50 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f794000b070 tx=0x7f7940004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:21.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f79457fa700 1 -- 172.21.15.115:0/3295507248 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794000e040 con 0x7f7948100090 2023-07-14T18:12:21.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794819c970 con 0x7f7948100090 2023-07-14T18:12:21.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f79457fa700 1 -- 172.21.15.115:0/3295507248 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f794001bcf0 con 0x7f7948100090 2023-07-14T18:12:21.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f79457fa700 1 -- 172.21.15.115:0/3295507248 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79400058d0 con 0x7f7948100090 2023-07-14T18:12:21.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.783+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794819ce20 con 0x7f7948100090 2023-07-14T18:12:21.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.784+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79481061c0 con 0x7f7948100090 2023-07-14T18:12:21.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.785+0000 7f79457fa700 1 -- 172.21.15.115:0/3295507248 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f794001b860 con 0x7f7948100090 2023-07-14T18:12:21.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.786+0000 7f79457fa700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79300a11e0 0x7f79300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:21.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.786+0000 7f7947fff700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79300a11e0 0x7f79300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:21.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.786+0000 7f79457fa700 1 -- 172.21.15.115:0/3295507248 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f79400cf290 con 0x7f7948100090 2023-07-14T18:12:21.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.786+0000 7f7947fff700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79300a11e0 0x7f79300a3690 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7948067520 tx=0x7f793c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:21.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.789+0000 7f79457fa700 1 -- 172.21.15.115:0/3295507248 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7940098500 con 0x7f7948100090 2023-07-14T18:12:22.022 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:21.981+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f794802c8e0 con 0x7f7948100090 2023-07-14T18:12:22.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 921 B/s wr, 7 op/s 2023-07-14T18:12:22.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T18:12:22.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T18:12:22.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: Cluster is now healthy 2023-07-14T18:12:22.360 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.187:6837/1000100433,v1:172.21.15.187:6839/1000100433] up:boot 2023-07-14T18:12:22.360 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.150:6836/3827925324,v1:172.21.15.150:6838/3827925324] up:boot 2023-07-14T18:12:22.360 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.187:6834/2984810379,v1:172.21.15.187:6835/2984810379] up:boot 2023-07-14T18:12:22.360 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.187:6836/4282111862,v1:172.21.15.187:6838/4282111862] up:boot 2023-07-14T18:12:22.361 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.150:6834/3797852532,v1:172.21.15.150:6835/3797852532] up:boot 2023-07-14T18:12:22.361 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: mds.? [v2:172.21.15.150:6837/743737414,v1:172.21.15.150:6839/743737414] up:boot 2023-07-14T18:12:22.361 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1444853747' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:12:22.361 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: fsmap 12 up:standby 2023-07-14T18:12:22.361 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:12:22.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:12:22.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:12:22.362 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:12:22.363 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:12:22.363 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:12:22.364 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:22 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3295507248' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:12:22.373 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 921 B/s wr, 7 op/s 2023-07-14T18:12:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T18:12:22.374 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T18:12:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: Cluster is now healthy 2023-07-14T18:12:22.375 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.187:6837/1000100433,v1:172.21.15.187:6839/1000100433] up:boot 2023-07-14T18:12:22.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.150:6836/3827925324,v1:172.21.15.150:6838/3827925324] up:boot 2023-07-14T18:12:22.376 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.187:6834/2984810379,v1:172.21.15.187:6835/2984810379] up:boot 2023-07-14T18:12:22.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.187:6836/4282111862,v1:172.21.15.187:6838/4282111862] up:boot 2023-07-14T18:12:22.377 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.150:6834/3797852532,v1:172.21.15.150:6835/3797852532] up:boot 2023-07-14T18:12:22.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: mds.? [v2:172.21.15.150:6837/743737414,v1:172.21.15.150:6839/743737414] up:boot 2023-07-14T18:12:22.378 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1444853747' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:12:22.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: fsmap 12 up:standby 2023-07-14T18:12:22.379 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:12:22.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:12:22.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:12:22.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:12:22.380 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:12:22.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:12:22.381 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:22 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3295507248' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:12:22.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 921 B/s wr, 7 op/s 2023-07-14T18:12:22.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T18:12:22.436 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T18:12:22.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: Cluster is now healthy 2023-07-14T18:12:22.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.187:6837/1000100433,v1:172.21.15.187:6839/1000100433] up:boot 2023-07-14T18:12:22.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.150:6836/3827925324,v1:172.21.15.150:6838/3827925324] up:boot 2023-07-14T18:12:22.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.187:6834/2984810379,v1:172.21.15.187:6835/2984810379] up:boot 2023-07-14T18:12:22.437 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.187:6836/4282111862,v1:172.21.15.187:6838/4282111862] up:boot 2023-07-14T18:12:22.438 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.150:6834/3797852532,v1:172.21.15.150:6835/3797852532] up:boot 2023-07-14T18:12:22.438 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: mds.? [v2:172.21.15.150:6837/743737414,v1:172.21.15.150:6839/743737414] up:boot 2023-07-14T18:12:22.438 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1444853747' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:12:22.438 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: fsmap 12 up:standby 2023-07-14T18:12:22.438 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:12:22.439 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:12:22.439 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:12:22.439 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:12:22.439 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:12:22.439 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:12:22.440 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:22 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3295507248' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:12:22.679 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.638+0000 7f79457fa700 1 -- 172.21.15.115:0/3295507248 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v91) v1 ==== 181+0+0 (secure 0 0 0) 0x7f794009bb20 con 0x7f7948100090 2023-07-14T18:12:22.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79300a11e0 msgr2=0x7f79300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:22.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79300a11e0 0x7f79300a3690 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7948067520 tx=0x7f793c008040 comp rx=0 tx=0).stop 2023-07-14T18:12:22.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7948100090 msgr2=0x7f794819be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:22.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7948100090 0x7f794819be50 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f794000b070 tx=0x7f7940004db0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 shutdown_connections 2023-07-14T18:12:22.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7f79300a11e0 0x7f79300a3690 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.684 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7948100090 0x7f794819be50 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79480fedc0 0x7f79481948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 --2- 172.21.15.115:0/3295507248 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f79480ff750 0x7f7948194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 >> 172.21.15.115:0/3295507248 conn(0x7f79480f98a0 msgr2=0x7f7948103300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:22.685 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 shutdown_connections 2023-07-14T18:12:22.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.641+0000 7f794e6ee700 1 -- 172.21.15.115:0/3295507248 wait complete. 2023-07-14T18:12:22.686 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_metadata' removed 2023-07-14T18:12:22.696 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-07-14T18:12:22.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 -- 172.21.15.115:0/3240771902 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd28101850 msgr2=0x7fdd281080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:22.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3240771902 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd28101850 0x7fdd281080f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000b0a0 tx=0x7fdd2000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 -- 172.21.15.115:0/3240771902 shutdown_connections 2023-07-14T18:12:22.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3240771902 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd28101850 0x7fdd281080f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3240771902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd280ffc10 0x7fdd28100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3240771902 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd28100ea0 0x7fdd28101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 -- 172.21.15.115:0/3240771902 >> 172.21.15.115:0/3240771902 conn(0x7fdd280fb210 msgr2=0x7fdd280fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:22.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 -- 172.21.15.115:0/3240771902 shutdown_connections 2023-07-14T18:12:22.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 -- 172.21.15.115:0/3240771902 wait complete. 2023-07-14T18:12:22.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.783+0000 7fdd2f226700 1 Processor -- start 2023-07-14T18:12:22.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2f226700 1 -- start start 2023-07-14T18:12:22.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2f226700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd280ffc10 0x7fdd28195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:22.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2f226700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd28100ea0 0x7fdd28195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:22.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2f226700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd28101850 0x7fdd2819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:22.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2f226700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2819d5b0 con 0x7fdd280ffc10 2023-07-14T18:12:22.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2f226700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2819d6f0 con 0x7fdd28101850 2023-07-14T18:12:22.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2f226700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2819d880 con 0x7fdd28100ea0 2023-07-14T18:12:22.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2cfc2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd280ffc10 0x7fdd28195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:22.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2cfc2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd280ffc10 0x7fdd28195a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54604/0 (socket says 172.21.15.115:54604) 2023-07-14T18:12:22.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2cfc2700 1 -- 172.21.15.115:0/3516909741 learned_addr learned my addr 172.21.15.115:0/3516909741 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:22.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2cfc2700 1 -- 172.21.15.115:0/3516909741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd28100ea0 msgr2=0x7fdd28195f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:22.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2cfc2700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd28100ea0 0x7fdd28195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2cfc2700 1 -- 172.21.15.115:0/3516909741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd28101850 msgr2=0x7fdd2819cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:22.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2cfc2700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd28101850 0x7fdd2819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:22.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2cfc2700 1 -- 172.21.15.115:0/3516909741 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20009d00 con 0x7fdd280ffc10 2023-07-14T18:12:22.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2cfc2700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd280ffc10 0x7fdd28195a30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00c920 tx=0x7fdd1c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:22.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd25ffb700 1 -- 172.21.15.115:0/3516909741 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c007b00 con 0x7fdd280ffc10 2023-07-14T18:12:22.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2819db60 con 0x7fdd280ffc10 2023-07-14T18:12:22.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd25ffb700 1 -- 172.21.15.115:0/3516909741 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdd1c004cc0 con 0x7fdd280ffc10 2023-07-14T18:12:22.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.784+0000 7fdd25ffb700 1 -- 172.21.15.115:0/3516909741 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c005490 con 0x7fdd280ffc10 2023-07-14T18:12:22.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.785+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2819e0c0 con 0x7fdd280ffc10 2023-07-14T18:12:22.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.785+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd28107240 con 0x7fdd280ffc10 2023-07-14T18:12:22.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.786+0000 7fdd25ffb700 1 -- 172.21.15.115:0/3516909741 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fdd1c004750 con 0x7fdd280ffc10 2023-07-14T18:12:22.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.787+0000 7fdd25ffb700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd180a11e0 0x7fdd180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:22.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.787+0000 7fdd25ffb700 1 -- 172.21.15.115:0/3516909741 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8724+0+0 (secure 0 0 0) 0x7fdd1c0ccce0 con 0x7fdd280ffc10 2023-07-14T18:12:22.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.787+0000 7fdd27fff700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd180a11e0 0x7fdd180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:22.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.788+0000 7fdd27fff700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd180a11e0 0x7fdd180a3690 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdd140097b0 tx=0x7fdd14006d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:22.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.791+0000 7fdd25ffb700 1 -- 172.21.15.115:0/3516909741 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdd1c0960e0 con 0x7fdd280ffc10 2023-07-14T18:12:23.021 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:22.981+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdd2802c8e0 con 0x7fdd280ffc10 2023-07-14T18:12:23.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.645+0000 7fdd25ffb700 1 -- 172.21.15.115:0/3516909741 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v92) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdd1c01d020 con 0x7fdd280ffc10 2023-07-14T18:12:23.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd180a11e0 msgr2=0x7fdd180a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:23.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd180a11e0 0x7fdd180a3690 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdd140097b0 tx=0x7fdd14006d20 comp rx=0 tx=0).stop 2023-07-14T18:12:23.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd280ffc10 msgr2=0x7fdd28195a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:23.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd280ffc10 0x7fdd28195a30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00c920 tx=0x7fdd1c00cce0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 shutdown_connections 2023-07-14T18:12:23.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fdd180a11e0 0x7fdd180a3690 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd28101850 0x7fdd2819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdd280ffc10 0x7fdd28195a30 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 --2- 172.21.15.115:0/3516909741 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd28100ea0 0x7fdd28195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 >> 172.21.15.115:0/3516909741 conn(0x7fdd280fb210 msgr2=0x7fdd28104380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 shutdown_connections 2023-07-14T18:12:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.649+0000 7fdd2f226700 1 -- 172.21.15.115:0/3516909741 wait complete. 2023-07-14T18:12:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_data' removed 2023-07-14T18:12:23.705 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-07-14T18:12:23.842 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.797+0000 7fe81328e700 1 -- 172.21.15.115:0/1724470657 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0ffd00 msgr2=0x7fe80c100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:23.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.797+0000 7fe81328e700 1 --2- 172.21.15.115:0/1724470657 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0ffd00 0x7fe80c100150 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe808009990 tx=0x7fe808009ca0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.843 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.797+0000 7fe81328e700 1 -- 172.21.15.115:0/1724470657 shutdown_connections 2023-07-14T18:12:23.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.797+0000 7fe81328e700 1 --2- 172.21.15.115:0/1724470657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80c100690 0x7fe80c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.844 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.797+0000 7fe81328e700 1 --2- 172.21.15.115:0/1724470657 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe80c0feb10 0x7fe80c0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.797+0000 7fe81328e700 1 --2- 172.21.15.115:0/1724470657 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0ffd00 0x7fe80c100150 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.797+0000 7fe81328e700 1 -- 172.21.15.115:0/1724470657 >> 172.21.15.115:0/1724470657 conn(0x7fe80c0fa130 msgr2=0x7fe80c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:23.845 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.797+0000 7fe81328e700 1 -- 172.21.15.115:0/1724470657 shutdown_connections 2023-07-14T18:12:23.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81328e700 1 -- 172.21.15.115:0/1724470657 wait complete. 2023-07-14T18:12:23.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81328e700 1 Processor -- start 2023-07-14T18:12:23.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81328e700 1 -- start start 2023-07-14T18:12:23.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81328e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0feb10 0x7fe80c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:23.846 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81328e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe80c0ffd00 0x7fe80c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:23.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81328e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80c100690 0x7fe80c19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:23.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81328e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c19d610 con 0x7fe80c0feb10 2023-07-14T18:12:23.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81328e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c19d780 con 0x7fe80c100690 2023-07-14T18:12:23.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81328e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c19d940 con 0x7fe80c0ffd00 2023-07-14T18:12:23.847 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81102a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0feb10 0x7fe80c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:23.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81102a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0feb10 0x7fe80c069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:54618/0 (socket says 172.21.15.115:54618) 2023-07-14T18:12:23.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe81102a700 1 -- 172.21.15.115:0/1380317256 learned_addr learned my addr 172.21.15.115:0/1380317256 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-07-14T18:12:23.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.798+0000 7fe810829700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe80c0ffd00 0x7fe80c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:23.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe81182b700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80c100690 0x7fe80c19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:23.848 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe81102a700 1 -- 172.21.15.115:0/1380317256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe80c0ffd00 msgr2=0x7fe80c19a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:23.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe81102a700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe80c0ffd00 0x7fe80c19a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe81102a700 1 -- 172.21.15.115:0/1380317256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80c100690 msgr2=0x7fe80c19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:23.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe81102a700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80c100690 0x7fe80c19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:23.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe81102a700 1 -- 172.21.15.115:0/1380317256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8080095f0 con 0x7fe80c0feb10 2023-07-14T18:12:23.849 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe81102a700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0feb10 0x7fe80c069340 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00b500 tx=0x7fe7fc00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:23.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe8027fc700 1 -- 172.21.15.115:0/1380317256 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc011830 con 0x7fe80c0feb10 2023-07-14T18:12:23.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe8027fc700 1 -- 172.21.15.115:0/1380317256 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe7fc011e20 con 0x7fe80c0feb10 2023-07-14T18:12:23.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c19dc20 con 0x7fe80c0feb10 2023-07-14T18:12:23.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe8027fc700 1 -- 172.21.15.115:0/1380317256 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc010dd0 con 0x7fe80c0feb10 2023-07-14T18:12:23.850 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.799+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c19e180 con 0x7fe80c0feb10 2023-07-14T18:12:23.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.800+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80c10a300 con 0x7fe80c0feb10 2023-07-14T18:12:23.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.801+0000 7fe8027fc700 1 -- 172.21.15.115:0/1380317256 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 19) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fe7fc011990 con 0x7fe80c0feb10 2023-07-14T18:12:23.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.802+0000 7fe8027fc700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a1260 0x7fe7f80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:12:23.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.802+0000 7fe810829700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a1260 0x7fe7f80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:12:23.851 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.802+0000 7fe8027fc700 1 -- 172.21.15.115:0/1380317256 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8252+0+0 (secure 0 0 0) 0x7fe7fc0cc360 con 0x7fe80c0feb10 2023-07-14T18:12:23.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.802+0000 7fe810829700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a1260 0x7fe7f80a3710 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe80800c010 tx=0x7fe808005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:12:23.852 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:23.806+0000 7fe8027fc700 1 -- 172.21.15.115:0/1380317256 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe7fc095a50 con 0x7fe80c0feb10 2023-07-14T18:12:23.936 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:23 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3295507248' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:12:23.936 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:23 smithi187 ceph-mon[129317]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T18:12:23.936 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:23 smithi187 ceph-mon[129317]: pgmap v318: 97 pgs: 97 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:12:23.936 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:23 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3516909741' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:12:24.048 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.007+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe80c10a4c0 con 0x7fe80c0feb10 2023-07-14T18:12:24.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:23 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3295507248' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:12:24.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:23 smithi115 ceph-mon[127451]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T18:12:24.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:23 smithi115 ceph-mon[127451]: pgmap v318: 97 pgs: 97 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:12:24.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:23 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3516909741' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:12:24.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:23 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3295507248' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:12:24.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:23 smithi150 ceph-mon[128853]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T18:12:24.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:23 smithi150 ceph-mon[128853]: pgmap v318: 97 pgs: 97 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:12:24.124 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:23 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3516909741' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:12:24.686 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.646+0000 7fe8027fc700 1 -- 172.21.15.115:0/1380317256 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v93) v1 ==== 178+0+0 (secure 0 0 0) 0x7fe7fc099070 con 0x7fe80c0feb10 2023-07-14T18:12:24.689 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a1260 msgr2=0x7fe7f80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a1260 0x7fe7f80a3710 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe80800c010 tx=0x7fe808005fb0 comp rx=0 tx=0).stop 2023-07-14T18:12:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0feb10 msgr2=0x7fe80c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:12:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0feb10 0x7fe80c069340 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00b500 tx=0x7fe7fc00b810 comp rx=0 tx=0).stop 2023-07-14T18:12:24.690 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 shutdown_connections 2023-07-14T18:12:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:6800/2625350855,v1:172.21.15.115:6801/2625350855] conn(0x7fe7f80a1260 0x7fe7f80a3710 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80c100690 0x7fe80c19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe80c0feb10 0x7fe80c069340 secure :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00b500 tx=0x7fe7fc00b810 comp rx=0 tx=0).stop 2023-07-14T18:12:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 --2- 172.21.15.115:0/1380317256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe80c0ffd00 0x7fe80c19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:12:24.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.649+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 >> 172.21.15.115:0/1380317256 conn(0x7fe80c0fa130 msgr2=0x7fe80c0fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:12:24.692 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.652+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 shutdown_connections 2023-07-14T18:12:24.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-07-14T18:12:24.652+0000 7fe81328e700 1 -- 172.21.15.115:0/1380317256 wait complete. 2023-07-14T18:12:24.693 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_data_ec' removed 2023-07-14T18:12:24.706 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:12:24.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:12:24.768 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:12:24.799 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-14T18:12:24.800 DEBUG:teuthology.orchestra.run.smithi115:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T18:12:24.817 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T18:12:24.834 DEBUG:teuthology.orchestra.run.smithi187:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T18:12:24.856 INFO:tasks.cephadm:Stopping all daemons... 2023-07-14T18:12:24.856 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-14T18:12:24.857 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.a 2023-07-14T18:12:25.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:24 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/3516909741' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:12:25.109 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:24 smithi115 ceph-mon[127451]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T18:12:25.109 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:24 smithi115 ceph-mon[127451]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:12:25.109 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:24 smithi115 ceph-mon[127451]: from='client.? 172.21.15.115:0/1380317256' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:12:25.109 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:24 smithi115 systemd[1]: Stopping Ceph mon.a for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:25.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:24 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/3516909741' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:12:25.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:24 smithi150 ceph-mon[128853]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T18:12:25.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:24 smithi150 ceph-mon[128853]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:12:25.123 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:24 smithi150 ceph-mon[128853]: from='client.? 172.21.15.115:0/1380317256' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:12:25.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:24 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/3516909741' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:12:25.186 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:24 smithi187 ceph-mon[129317]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T18:12:25.186 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:24 smithi187 ceph-mon[129317]: from='mgr.14150 172.21.15.115:0/3319619686' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:12:25.186 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:24 smithi187 ceph-mon[129317]: from='client.? 172.21.15.115:0/1380317256' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:12:25.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-a[127427]: 2023-07-14T18:12:25.417+0000 7f2988ad3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:12:25.730 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-a[127427]: 2023-07-14T18:12:25.417+0000 7f2988ad3700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-14T18:12:26.076 INFO:journalctl@ceph.mon.a.smithi115.stdout:Jul 14 18:12:25 smithi115 bash[179050]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-a 2023-07-14T18:12:26.331 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.a.service' 2023-07-14T18:12:26.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:12:26.378 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-14T18:12:26.379 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-14T18:12:26.379 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.b 2023-07-14T18:12:26.737 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:26 smithi150 systemd[1]: Stopping Ceph mon.b for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:27.084 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:26 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-b[128829]: 2023-07-14T18:12:26.696+0000 7f8728f16700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:12:27.085 INFO:journalctl@ceph.mon.b.smithi150.stdout:Jul 14 18:12:26 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-b[128829]: 2023-07-14T18:12:26.696+0000 7f8728f16700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-14T18:12:27.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.b.service' 2023-07-14T18:12:27.336 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:12:27.336 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-14T18:12:27.337 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-14T18:12:27.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.c 2023-07-14T18:12:27.685 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:27 smithi187 systemd[1]: Stopping Ceph mon.c for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:28.035 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-c[129290]: 2023-07-14T18:12:27.662+0000 7f3e60866700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:12:28.036 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-c[129290]: 2023-07-14T18:12:27.662+0000 7f3e60866700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-14T18:12:28.406 INFO:journalctl@ceph.mon.c.smithi187.stdout:Jul 14 18:12:28 smithi187 bash[157501]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mon-c 2023-07-14T18:12:28.413 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mon.c.service' 2023-07-14T18:12:28.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:12:28.470 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-14T18:12:28.470 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-14T18:12:28.471 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T18:12:28.833 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:12:28 smithi115 systemd[1]: Stopping Ceph mgr.x for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:29.111 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:12:28 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x[127694]: 2023-07-14T18:12:28.791+0000 7f466a174700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:12:29.364 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Jul 14 18:12:29 smithi115 bash[179338]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-x 2023-07-14T18:12:29.370 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.x.service' 2023-07-14T18:12:29.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:12:29.409 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-14T18:12:29.410 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-14T18:12:29.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.y 2023-07-14T18:12:29.839 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:12:29 smithi150 systemd[1]: Stopping Ceph mgr.y for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:30.123 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:12:29 smithi150 bash[156284]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-y 2023-07-14T18:12:30.123 INFO:journalctl@ceph.mgr.y.smithi150.stdout:Jul 14 18:12:29 smithi150 systemd[1]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-14T18:12:30.201 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.y.service' 2023-07-14T18:12:30.237 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:12:30.238 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-14T18:12:30.238 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-14T18:12:30.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.z 2023-07-14T18:12:30.666 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:12:30 smithi187 systemd[1]: Stopping Ceph mgr.z for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:30.936 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:12:30 smithi187 bash[157656]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mgr-z 2023-07-14T18:12:31.130 INFO:journalctl@ceph.mgr.z.smithi187.stdout:Jul 14 18:12:30 smithi187 systemd[1]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-14T18:12:31.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mgr.z.service' 2023-07-14T18:12:31.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:12:31.175 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-14T18:12:31.176 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-14T18:12:31.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.0 2023-07-14T18:12:31.533 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:12:31 smithi115 systemd[1]: Stopping Ceph osd.0 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:31.858 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:12:31 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-0[140360]: 2023-07-14T18:12:31.491+0000 7efcce10b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:12:31.858 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:12:31 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-0[140360]: 2023-07-14T18:12:31.491+0000 7efcce10b700 -1 osd.0 93 *** Got signal Terminated *** 2023-07-14T18:12:31.859 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:12:31 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-0[140360]: 2023-07-14T18:12:31.491+0000 7efcce10b700 -1 osd.0 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:12:37.608 INFO:journalctl@ceph.osd.0.smithi115.stdout:Jul 14 18:12:37 smithi115 bash[179479]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-0 2023-07-14T18:12:38.802 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.0.service' 2023-07-14T18:12:38.843 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:12:38.843 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-14T18:12:38.844 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-14T18:12:38.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.1 2023-07-14T18:12:39.126 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:12:38 smithi150 systemd[1]: Stopping Ceph osd.1 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:39.873 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:12:39 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-1[133833]: 2023-07-14T18:12:39.359+0000 7fa8ba5dc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:12:39.874 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:12:39 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-1[133833]: 2023-07-14T18:12:39.359+0000 7fa8ba5dc700 -1 osd.1 93 *** Got signal Terminated *** 2023-07-14T18:12:39.874 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:12:39 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-1[133833]: 2023-07-14T18:12:39.359+0000 7fa8ba5dc700 -1 osd.1 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:12:45.624 INFO:journalctl@ceph.osd.1.smithi150.stdout:Jul 14 18:12:45 smithi150 bash[156426]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-1 2023-07-14T18:12:46.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.1.service' 2023-07-14T18:12:46.487 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:12:46.487 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-14T18:12:46.488 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-14T18:12:46.489 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.2 2023-07-14T18:12:46.858 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:12:46 smithi187 systemd[1]: Stopping Ceph osd.2 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:47.186 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:12:46 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-2[135131]: 2023-07-14T18:12:46.816+0000 7fc69e705700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:12:47.186 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:12:46 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-2[135131]: 2023-07-14T18:12:46.816+0000 7fc69e705700 -1 osd.2 93 *** Got signal Terminated *** 2023-07-14T18:12:47.187 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:12:46 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-2[135131]: 2023-07-14T18:12:46.816+0000 7fc69e705700 -1 osd.2 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:12:52.529 INFO:journalctl@ceph.osd.2.smithi187.stdout:Jul 14 18:12:52 smithi187 bash[157811]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-2 2023-07-14T18:12:53.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.2.service' 2023-07-14T18:12:53.835 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:12:53.836 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-14T18:12:53.837 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-14T18:12:53.838 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.3 2023-07-14T18:12:54.237 INFO:journalctl@ceph.osd.3.smithi115.stdout:Jul 14 18:12:53 smithi115 systemd[1]: Stopping Ceph osd.3 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:12:54.608 INFO:journalctl@ceph.osd.3.smithi115.stdout:Jul 14 18:12:54 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-3[145807]: 2023-07-14T18:12:54.195+0000 7f1326fed700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:12:54.609 INFO:journalctl@ceph.osd.3.smithi115.stdout:Jul 14 18:12:54 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-3[145807]: 2023-07-14T18:12:54.195+0000 7f1326fed700 -1 osd.3 93 *** Got signal Terminated *** 2023-07-14T18:12:54.609 INFO:journalctl@ceph.osd.3.smithi115.stdout:Jul 14 18:12:54 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-3[145807]: 2023-07-14T18:12:54.195+0000 7f1326fed700 -1 osd.3 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:12:57.858 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:12:57 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:12:57.364+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:12:58.734 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:12:58 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:12:58.364+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:12:59.108 INFO:journalctl@ceph.osd.3.smithi115.stdout:Jul 14 18:12:58 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-3[145807]: 2023-07-14T18:12:58.692+0000 7f131f5dc700 -1 osd.3 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.869175+0000 front 2023-07-14T18:12:34.869198+0000 (oldest deadline 2023-07-14T18:12:57.670145+0000) 2023-07-14T18:12:59.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:12:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:12:58.922+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:12:59.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:12:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:12:58.982+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:12:59.858 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:12:59 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:12:59.353+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:00.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:12:59 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:12:59.621+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:00.358 INFO:journalctl@ceph.osd.3.smithi115.stdout:Jul 14 18:12:59 smithi115 bash[179763]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-3 2023-07-14T18:13:00.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:12:59 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:12:59.961+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:00.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:12:59 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:12:59.938+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:00.687 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:00 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:00.325+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:01.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:00 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:00.645+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:01.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:00 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:00.981+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:01.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:00 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:00.913+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:01.472 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.3.service' 2023-07-14T18:13:01.513 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:13:01.513 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-14T18:13:01.514 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-14T18:13:01.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.4 2023-07-14T18:13:01.858 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:01 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:01.357+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:01.873 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:13:01 smithi150 systemd[1]: Stopping Ceph osd.4 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:13:01.874 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:01 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:01.635+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:02.124 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:13:01 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-4[139346]: 2023-07-14T18:13:01.832+0000 7f2f20522700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:13:02.124 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:13:01 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-4[139346]: 2023-07-14T18:13:01.832+0000 7f2f20522700 -1 osd.4 93 *** Got signal Terminated *** 2023-07-14T18:13:02.124 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:13:01 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-4[139346]: 2023-07-14T18:13:01.832+0000 7f2f20522700 -1 osd.4 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:13:02.186 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:02 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:02.029+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:02.187 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:01 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:01.869+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:02.858 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:02 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:02.389+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:03.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:02 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:02.660+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:03.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:03 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:03.010+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:03.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:02 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:02.899+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:03.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:03 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:03.434+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:04.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:03 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:03.684+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:04.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:04 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:03.999+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:04.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:03 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:03.941+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:04.858 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:04 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:04.475+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:05.123 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:04 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:04.692+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:05.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:05 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:05.045+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:05.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:04 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:04.921+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:05.858 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:05 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:05.479+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:06.124 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:13:05 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-4[139346]: 2023-07-14T18:13:05.748+0000 7f2f1ab15700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:39.608651+0000 front 2023-07-14T18:12:39.608620+0000 (oldest deadline 2023-07-14T18:13:04.908785+0000) 2023-07-14T18:13:06.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:05 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:05.734+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:06.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:05 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:05.734+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:06.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:06 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:06.073+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:06.437 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:06 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:06.073+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:42.204673+0000 front 2023-07-14T18:12:42.204843+0000 (oldest deadline 2023-07-14T18:13:05.104887+0000) 2023-07-14T18:13:06.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:05 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:05.911+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:06.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:05 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:05.911+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:06.438 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:06 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:06.133+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:06.858 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:06 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:06.388+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:06.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:06 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:06.451+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:06.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:06 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:06.451+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:07.045 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:13:06 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-4[139346]: 2023-07-14T18:13:06.780+0000 7f2f1ab15700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:39.608651+0000 front 2023-07-14T18:12:39.608620+0000 (oldest deadline 2023-07-14T18:13:04.908785+0000) 2023-07-14T18:13:07.045 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:06 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:06.753+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:07.045 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:06 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:06.753+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:07.374 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:07 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:07.003+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:07.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:07 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:07.086+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:07.437 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:07 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:07.086+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:42.204673+0000 front 2023-07-14T18:12:42.204843+0000 (oldest deadline 2023-07-14T18:13:05.104887+0000) 2023-07-14T18:13:07.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:07 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:07.150+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:07.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:06 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:06.922+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:07.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:06 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:06.922+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:07.831 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:13:07 smithi150 bash[156700]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-4 2023-07-14T18:13:07.858 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:07 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:07.423+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:07.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:07 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:07.426+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:07.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:07 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:07.426+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:08.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:07 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:07.994+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:08.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:07 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:07.789+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:08.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:07 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:07.789+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:08.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:08 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:08.049+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:08.437 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:08 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:08.049+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:42.204673+0000 front 2023-07-14T18:12:42.204843+0000 (oldest deadline 2023-07-14T18:13:05.104887+0000) 2023-07-14T18:13:08.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:08 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:08.118+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:08.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:07 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:07.964+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:08.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:07 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:07.964+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:08.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.4.service' 2023-07-14T18:13:08.654 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:13:08 smithi150 systemd[1]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.4.service: Succeeded. 2023-07-14T18:13:08.654 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jul 14 18:13:08 smithi150 systemd[1]: Stopped Ceph osd.4 for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:13:08.667 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:13:08.668 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-14T18:13:08.668 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-14T18:13:08.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.5 2023-07-14T18:13:08.858 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:08 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:08.392+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:08.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:08 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:08.387+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:08.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:08 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:08.387+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:08.973 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:08 smithi187 systemd[1]: Stopping Ceph osd.5 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:13:08.974 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:08 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:08.931+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:08.974 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:08 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:08.931+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:09.072 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:08 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:08.746+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:09.072 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:08 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:08.746+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:09.374 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:09 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:09.031+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:09.436 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:09 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:09.073+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:09.437 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:08 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:08.995+0000 7f865dacd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:13:09.438 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:08 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:08.995+0000 7f865dacd700 -1 osd.5 93 *** Got signal Terminated *** 2023-07-14T18:13:09.438 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:08 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:08.995+0000 7f865dacd700 -1 osd.5 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:13:09.439 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:09 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:09.094+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:09.439 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:09 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:09.094+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:42.204673+0000 front 2023-07-14T18:12:42.204843+0000 (oldest deadline 2023-07-14T18:13:05.104887+0000) 2023-07-14T18:13:09.858 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:09 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:09.353+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:09.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:09 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:09.368+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:09.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:09 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:09.368+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:10.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:09 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:09.992+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:10.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:09 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:09.794+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:10.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:09 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:09.794+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:10.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:10 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:10.083+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:10.437 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:10 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:10.083+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:42.204673+0000 front 2023-07-14T18:12:42.204843+0000 (oldest deadline 2023-07-14T18:13:05.104887+0000) 2023-07-14T18:13:10.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:10 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:10.105+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:10.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:09 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:09.966+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:10.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:09 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:09.966+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:10.858 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:10 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:10.330+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:10.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:10 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:10.386+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:10.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:10 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:10.386+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:11.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:10 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:10.956+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:11.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:10 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:10.808+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:11.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:10 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:10.808+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:11.436 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:11 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:11.059+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:11.437 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:11 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:11.110+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:11.437 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:11 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:11.110+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:42.204673+0000 front 2023-07-14T18:12:42.204843+0000 (oldest deadline 2023-07-14T18:13:05.104887+0000) 2023-07-14T18:13:11.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:10 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:10.982+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:11.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:10 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:10.982+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:11.608 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:11.283+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:11.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:11.375+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:11.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:11 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:11.375+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:12.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:12 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:11.999+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:12.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:11 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:11.788+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:12.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:11 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:11.788+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:12.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:12 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:12.091+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:12.437 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:12 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:12.091+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:42.204673+0000 front 2023-07-14T18:12:42.204843+0000 (oldest deadline 2023-07-14T18:13:05.104887+0000) 2023-07-14T18:13:12.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:12 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:12.069+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:12.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:11 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:11.989+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:12.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:11 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:11.989+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:12.608 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:12 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:12.259+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:12.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:12 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:12.348+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:12.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:12 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:12.348+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:13.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:13 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:13.011+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:13.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:12 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:12.766+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:13.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:12 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:12.766+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:13.436 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:13 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:13.060+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:32.203333+0000 front 2023-07-14T18:12:32.203411+0000 (oldest deadline 2023-07-14T18:12:58.103424+0000) 2023-07-14T18:13:13.437 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:13 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5[140647]: 2023-07-14T18:13:13.060+0000 7f86560bc700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:42.204673+0000 front 2023-07-14T18:12:42.204843+0000 (oldest deadline 2023-07-14T18:13:05.104887+0000) 2023-07-14T18:13:13.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:13 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:13.028+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:13.438 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:13 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:13.028+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:13.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:13 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:13.015+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:13.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:13 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:13.015+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:13.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:13 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:13.015+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:13.608 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:13 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:13.263+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:13.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:13 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:13.393+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:13.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:13 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:13.393+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:14.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:13 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:13.973+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:14.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:13 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:13.750+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:14.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:13 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:13.750+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:14.436 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:14 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:14.001+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:14.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:14 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:14.001+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:14.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:14 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:14.056+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:14.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:14 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:14.056+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:14.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:14 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:14.056+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:14.609 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:14 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:14.296+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:14.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:14 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:14.425+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:14.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:14 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:14.425+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:14.610 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:14 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:14.425+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:14.705 INFO:journalctl@ceph.osd.5.smithi187.stdout:Jul 14 18:13:14 smithi187 bash[158071]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-5 2023-07-14T18:13:15.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:14 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:14.957+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:15.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:14 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:14.788+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:15.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:14 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:14.788+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:15.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:14 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:14.788+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:15.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:15 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:15.026+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:15.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:15 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:15.026+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:15.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:15 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:15.076+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:15.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:15 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:15.076+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:15.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:15 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:15.076+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:15.608 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:15 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:15.264+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:15.609 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:15 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:15.264+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.268204+0000 front 2023-07-14T18:12:51.268219+0000 (oldest deadline 2023-07-14T18:13:14.768124+0000) 2023-07-14T18:13:15.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:15 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:15.472+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:15.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:15 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:15.472+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:15.610 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:15 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:15.472+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:16.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.5.service' 2023-07-14T18:13:16.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:13:16.121 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-14T18:13:16.121 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-14T18:13:16.121 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.6 2023-07-14T18:13:16.123 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:15 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:15.958+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:16.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:15 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:15.827+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:16.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:15 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:15.827+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:16.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:15 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:15.827+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:16.436 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:16 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:16.016+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:16.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:16 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:16.016+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:16.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:16 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:16.103+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:16.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:16 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:16.104+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:16.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:16 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:16.104+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:16.472 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:16 smithi115 systemd[1]: Stopping Ceph osd.6 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:13:16.473 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:16 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:16.274+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:16.473 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:16 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:16.274+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.268204+0000 front 2023-07-14T18:12:51.268219+0000 (oldest deadline 2023-07-14T18:13:14.768124+0000) 2023-07-14T18:13:16.473 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:16 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:16.430+0000 7f5b97df7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:13:16.859 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:16 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:16.430+0000 7f5b97df7700 -1 osd.6 93 *** Got signal Terminated *** 2023-07-14T18:13:16.859 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:16 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:16.430+0000 7f5b97df7700 -1 osd.6 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:13:16.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:16 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:16.496+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:16.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:16 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:16.496+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:16.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:16 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:16.496+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:17.374 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:16 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:16.940+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:17.374 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:16 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:16.940+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:17.374 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:16 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:16.864+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:17.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:16 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:16.864+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:17.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:16 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:16.864+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:17.436 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:17 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:17.013+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:17.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:17 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:17.013+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:17.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:17 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:17.116+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:17.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:17 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:17.116+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:17.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:17 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:17.116+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:17.544 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:17 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:17.252+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:17.544 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:17 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:17.252+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.268204+0000 front 2023-07-14T18:12:51.268219+0000 (oldest deadline 2023-07-14T18:13:14.768124+0000) 2023-07-14T18:13:17.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:17 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:17.502+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:17.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:17 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:17.502+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:17.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:17 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:17.502+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:18.374 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:17 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:17.908+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:18.374 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:17 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:17.908+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:18.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:17 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:17.836+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:18.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:17 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:17.836+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:18.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:17 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:17.836+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:18.436 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:18 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:18.005+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:18.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:18 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:18.005+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:18.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:18 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:18.157+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:18.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:18 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:18.157+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:18.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:18 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:18.157+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:18.568 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:18 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:18.242+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:18.568 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:18 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:18.242+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.268204+0000 front 2023-07-14T18:12:51.268219+0000 (oldest deadline 2023-07-14T18:13:14.768124+0000) 2023-07-14T18:13:18.858 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:18 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:18.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:18.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:18 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:18.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:18.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:18 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:18.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:19.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:18 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:18.944+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:19.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:18 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:18.944+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:19.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:18 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:18.791+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:19.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:18 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:18.791+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:19.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:18 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:18.791+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:19.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:18 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:18.987+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:19.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:18 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:18.987+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:19.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:19 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:19.197+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:19.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:19 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:19.197+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:19.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:19 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:19.197+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:19.561 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:19 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:19.222+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:19.562 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:19 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:19.222+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.268204+0000 front 2023-07-14T18:12:51.268219+0000 (oldest deadline 2023-07-14T18:13:14.768124+0000) 2023-07-14T18:13:19.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:19 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:19.519+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:19.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:19 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:19.519+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:19.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:19 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:19.519+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:20.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:19 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:19.943+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:20.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:19 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:19.943+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:20.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:19 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:19.747+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:20.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:19 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:19.747+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:20.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:19 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:19.747+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:20.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:19 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:19.961+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:20.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:19 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:19.961+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:20.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:20.157+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:20.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:20.157+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:20.439 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:20.157+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:20.524 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:20 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:20.210+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:20.524 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:20 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:20.210+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.268204+0000 front 2023-07-14T18:12:51.268219+0000 (oldest deadline 2023-07-14T18:13:14.768124+0000) 2023-07-14T18:13:20.524 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:20 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:20.210+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:54.768511+0000 front 2023-07-14T18:12:54.768490+0000 (oldest deadline 2023-07-14T18:13:20.068749+0000) 2023-07-14T18:13:20.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:20 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:20.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:20.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:20 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:20.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:20.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:20 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:20.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:21.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:20.938+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:21.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:20.938+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:21.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:20.938+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.542065+0000 front 2023-07-14T18:12:56.542025+0000 (oldest deadline 2023-07-14T18:13:20.041886+0000) 2023-07-14T18:13:21.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:20.791+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:21.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:20.791+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:21.127 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:20.791+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:21.127 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:20.791+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:21.436 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:20.957+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:21.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:20 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:20.957+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:21.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:21 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:21.126+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:21.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:21 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:21.126+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:21.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:21 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:21.126+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:21.549 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:21 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:21.231+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.965907+0000 front 2023-07-14T18:12:41.965950+0000 (oldest deadline 2023-07-14T18:13:05.466117+0000) 2023-07-14T18:13:21.549 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:21 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:21.231+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.268204+0000 front 2023-07-14T18:12:51.268219+0000 (oldest deadline 2023-07-14T18:13:14.768124+0000) 2023-07-14T18:13:21.549 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:21 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6[151363]: 2023-07-14T18:13:21.231+0000 7f5b90be7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:54.768511+0000 front 2023-07-14T18:12:54.768490+0000 (oldest deadline 2023-07-14T18:13:20.068749+0000) 2023-07-14T18:13:21.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:21 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:21.506+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:21.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:21 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:21.506+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:21.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:21 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:21.507+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:21.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:21 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:21.507+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:22.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:21 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:21.941+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:22.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:21 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:21.941+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:22.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:21 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:21.941+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.542065+0000 front 2023-07-14T18:12:56.542025+0000 (oldest deadline 2023-07-14T18:13:20.041886+0000) 2023-07-14T18:13:22.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:21 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:21.771+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:22.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:21 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:21.771+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:22.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:21 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:21.771+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:22.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:21 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:21.771+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:22.359 INFO:journalctl@ceph.osd.6.smithi115.stdout:Jul 14 18:13:22 smithi115 bash[180030]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-6 2023-07-14T18:13:22.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:21 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:21.951+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:22.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:21 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:21.951+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:22.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:22 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:22.171+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:22.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:22 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:22.171+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:22.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:22 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:22.171+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:22.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:22.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:22.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:22.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:22.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:22.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:22.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:22 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:22.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:23.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:22 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:22.971+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:23.128 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:22 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:22.971+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:23.128 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:22 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:22.971+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.542065+0000 front 2023-07-14T18:12:56.542025+0000 (oldest deadline 2023-07-14T18:13:20.041886+0000) 2023-07-14T18:13:23.128 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:22 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:22.726+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:23.129 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:22 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:22.726+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:23.129 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:22 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:22.726+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:23.129 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:22 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:22.726+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:23.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:22 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:22.912+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:23.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:22 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:22.912+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:23.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:22 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:22.912+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:23.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:23 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:23.147+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:23.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:23 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:23.147+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:23.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:23 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:23.147+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:23.609 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:23 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:23.492+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:23.610 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:23 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:23.492+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:23.610 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:23 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:23.492+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:23.611 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:23 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:23.492+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:23.720 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.6.service' 2023-07-14T18:13:23.760 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:13:23.761 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-14T18:13:23.761 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-14T18:13:23.762 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.7 2023-07-14T18:13:24.048 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:23 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:23.766+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:24.048 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:23 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:23.766+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:24.048 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:23 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:23.766+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:24.049 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:23 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:23.766+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:24.049 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:23 smithi150 systemd[1]: Stopping Ceph osd.7 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:13:24.374 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:24.006+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:24.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:24.006+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:24.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:24.006+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.542065+0000 front 2023-07-14T18:12:56.542025+0000 (oldest deadline 2023-07-14T18:13:20.041886+0000) 2023-07-14T18:13:24.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:24.063+0000 7f86ecf9e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:13:24.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:24.063+0000 7f86ecf9e700 -1 osd.7 93 *** Got signal Terminated *** 2023-07-14T18:13:24.375 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:24.063+0000 7f86ecf9e700 -1 osd.7 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:13:24.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:23 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:23.953+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:24.437 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:23 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:23.953+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:24.438 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:23 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:23.953+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:24.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:24 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:24.189+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:24.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:24 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:24.189+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:24.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:24 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:24.189+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:24.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:24 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:24.463+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:24.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:24 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:24.463+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:24.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:24 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:24.463+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:24.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:24 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:24.463+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:25.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:24.792+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:25.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:24.792+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:25.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:24.792+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:25.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:24.792+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:25.127 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:24.989+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:25.127 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:24.989+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:25.127 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:24 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:24.989+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.542065+0000 front 2023-07-14T18:12:56.542025+0000 (oldest deadline 2023-07-14T18:13:20.041886+0000) 2023-07-14T18:13:25.266 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:24 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:24.928+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:25.267 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:24 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:24.928+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:25.267 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:24 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:24.928+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:25.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:25 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:25.224+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:25.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:25 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:25.224+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:25.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:25 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:25.224+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:25.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:25.435+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:25.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:25.435+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:25.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:25.435+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:25.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:25 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:25.435+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:26.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:25 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:25.966+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:26.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:25 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:25.966+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:26.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:25 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:25.966+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.542065+0000 front 2023-07-14T18:12:56.542025+0000 (oldest deadline 2023-07-14T18:13:20.041886+0000) 2023-07-14T18:13:26.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:25 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:25.820+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:26.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:25 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:25.820+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:26.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:25 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:25.820+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:26.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:25 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:25.820+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:26.233 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:25 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:25.917+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:26.234 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:25 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:25.917+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:26.234 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:25 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:25.917+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:26.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:26.191+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:26.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:26.191+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:26.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:26.191+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:26.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:26 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:26.404+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:26.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:26 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:26.404+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:26.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:26 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:26.404+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:26.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:26 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:26.404+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:27.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:27.000+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:27.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:27.000+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:27.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:27.000+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.542065+0000 front 2023-07-14T18:12:56.542025+0000 (oldest deadline 2023-07-14T18:13:20.041886+0000) 2023-07-14T18:13:27.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:26 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:26.772+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:27.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:26 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:26.772+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:27.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:26 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:26.772+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:27.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:26 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:26.772+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:27.282 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:26.911+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:27.282 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:26.911+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:27.283 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:26 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:26.911+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:27.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:27.240+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:27.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:27.240+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:27.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:27.240+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:27.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:27.443+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:27.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:27.443+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:27.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:27.443+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:27.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:27.443+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:27.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:27 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:27.443+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:28.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:27.972+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:28.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:27.972+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:28.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:27.972+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.542065+0000 front 2023-07-14T18:12:56.542025+0000 (oldest deadline 2023-07-14T18:13:20.041886+0000) 2023-07-14T18:13:28.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:27.744+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:28.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:27.744+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:28.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:27.744+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:28.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:27 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:27.744+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:28.311 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:27.953+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:28.311 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:27.953+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:28.311 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:27.953+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:28.311 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:27 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:27.953+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.497136+0000 front 2023-07-14T18:13:06.497277+0000 (oldest deadline 2023-07-14T18:13:27.597080+0000) 2023-07-14T18:13:28.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:28.268+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:28.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:28.268+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:28.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:28.268+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:28.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:28 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:28.429+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:28.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:28 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:28.429+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:28.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:28 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:28.429+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:28.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:28 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:28.429+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:28.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:28 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:28.429+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:29.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:28.923+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.339670+0000 front 2023-07-14T18:12:41.339636+0000 (oldest deadline 2023-07-14T18:13:06.039787+0000) 2023-07-14T18:13:29.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:28.923+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:48.940977+0000 front 2023-07-14T18:12:48.940811+0000 (oldest deadline 2023-07-14T18:13:16.541307+0000) 2023-07-14T18:13:29.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:28.923+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.542065+0000 front 2023-07-14T18:12:56.542025+0000 (oldest deadline 2023-07-14T18:13:20.041886+0000) 2023-07-14T18:13:29.125 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7[144709]: 2023-07-14T18:13:28.923+0000 7f86e7591700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.742821+0000 front 2023-07-14T18:13:04.742794+0000 (oldest deadline 2023-07-14T18:13:28.843014+0000) 2023-07-14T18:13:29.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:28.696+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:29.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:28.696+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:29.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:28.696+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:29.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:28.696+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:29.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:28 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:28.696+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:29.320 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:28.976+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:29.321 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:28.976+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:29.321 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:28.976+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:29.321 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:28 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:28.976+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.497136+0000 front 2023-07-14T18:13:06.497277+0000 (oldest deadline 2023-07-14T18:13:27.597080+0000) 2023-07-14T18:13:29.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:29.278+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:29.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:29.278+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:29.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:29.278+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:29.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:29.278+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:29.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:29 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:29.411+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:29.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:29 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:29.411+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:29.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:29 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:29.411+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:29.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:29 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:29.411+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:29.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:29 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:29.411+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:30.124 INFO:journalctl@ceph.osd.7.smithi150.stdout:Jul 14 18:13:29 smithi150 bash[156965]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-7 2023-07-14T18:13:30.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:29.716+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:30.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:29.716+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:30.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:29.716+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:30.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:29.716+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:30.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:29.716+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:30.326 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:29.931+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:30.327 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:29.931+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:30.327 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:29.931+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:30.327 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:29 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:29.931+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.497136+0000 front 2023-07-14T18:13:06.497277+0000 (oldest deadline 2023-07-14T18:13:27.597080+0000) 2023-07-14T18:13:30.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:30.284+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:30.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:30.284+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:30.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:30.284+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:30.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:30.284+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:30.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:30 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:30.434+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:30.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:30 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:30.434+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:30.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:30 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:30.434+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:30.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:30 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:30.434+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:30.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:30 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:30.434+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:30.969 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:30 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:30.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:30.969 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:30 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:30.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:30.969 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:30 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:30.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:30.969 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:30 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:30.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:30.970 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:30 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:30.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:30.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.7.service' 2023-07-14T18:13:31.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:13:31.015 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-14T18:13:31.016 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-14T18:13:31.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.8 2023-07-14T18:13:31.189 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:30.885+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:31.189 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:30.885+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:31.190 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:30.885+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:31.190 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:30 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:30.885+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.497136+0000 front 2023-07-14T18:13:06.497277+0000 (oldest deadline 2023-07-14T18:13:27.597080+0000) 2023-07-14T18:13:31.190 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:31 smithi187 systemd[1]: Stopping Ceph osd.8 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:13:31.687 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:31.333+0000 7fcb7375c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:13:31.688 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:31.333+0000 7fcb7375c700 -1 osd.8 93 *** Got signal Terminated *** 2023-07-14T18:13:31.688 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:31.333+0000 7fcb7375c700 -1 osd.8 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:13:31.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:31.245+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:31.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:31.245+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:31.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:31.245+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:31.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:31.245+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:31.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:31 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:31.457+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:31.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:31 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:31.457+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:31.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:31 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:31.457+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:31.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:31 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:31.457+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:31.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:31 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:31.457+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:32.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:31 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:31.808+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:32.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:31 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:31.808+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:32.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:31 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:31.808+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:32.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:31 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:31.808+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:32.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:31 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:31.808+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:32.322 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:31.913+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:32.323 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:31.913+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:32.323 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:31.913+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:32.323 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:31 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:31.913+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.497136+0000 front 2023-07-14T18:13:06.497277+0000 (oldest deadline 2023-07-14T18:13:27.597080+0000) 2023-07-14T18:13:32.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:32.280+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:32.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:32.280+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:32.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:32.280+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:32.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:32.280+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:32.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:32 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:32.441+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:32.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:32 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:32.441+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:32.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:32 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:32.441+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:32.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:32 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:32.441+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:32.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:32 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:32.441+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:33.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:32 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:32.823+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:33.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:32 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:32.823+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:33.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:32 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:32.823+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:33.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:32 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:32.823+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:33.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:32 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:32.823+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:33.346 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:32.924+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:33.346 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:32.924+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:33.347 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:32.924+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:33.347 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:32 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:32.924+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.497136+0000 front 2023-07-14T18:13:06.497277+0000 (oldest deadline 2023-07-14T18:13:27.597080+0000) 2023-07-14T18:13:33.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:33 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:33.304+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:33.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:33 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:33.304+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:33.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:33 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:33.304+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:33.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:33 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:33.304+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:33.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:33 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:33.466+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:33.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:33 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:33.466+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:33.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:33 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:33.466+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:33.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:33 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:33.466+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:33.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:33 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:33.466+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:34.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:33.807+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:34.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:33.807+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:34.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:33.807+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:34.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:33.807+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:34.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:33.807+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:34.359 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:33 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:33.907+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:34.359 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:33 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:33.907+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:34.359 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:33 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:33.907+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:34.360 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:33 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:33.907+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.497136+0000 front 2023-07-14T18:13:06.497277+0000 (oldest deadline 2023-07-14T18:13:27.597080+0000) 2023-07-14T18:13:34.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:34 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:34.316+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:34.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:34 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:34.316+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:34.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:34 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:34.316+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:34.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:34 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:34.316+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:34.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:34 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:34.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:34.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:34 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:34.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:34.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:34 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:34.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:34.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:34 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:34.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:34.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:34 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:34.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:35.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:34 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:34.809+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:35.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:34 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:34.809+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:35.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:34 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:34.809+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:35.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:34 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:34.809+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:35.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:34 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:34.809+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:35.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:34 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:34.809+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:35.187 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:34 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:34.880+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:35.187 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:34 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:34.880+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:35.187 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:34 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:34.880+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:35.188 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:34 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:34.880+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.497136+0000 front 2023-07-14T18:13:06.497277+0000 (oldest deadline 2023-07-14T18:13:27.597080+0000) 2023-07-14T18:13:35.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:35.279+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:35.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:35.279+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:35.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:35.279+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:35.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:35.279+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:35.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:35 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:35.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:35.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:35 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:35.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:35.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:35 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:35.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:35.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:35 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:35.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:35.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:35 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:35.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:35.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:35 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:35.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.595922+0000 front 2023-07-14T18:13:12.596059+0000 (oldest deadline 2023-07-14T18:13:34.895982+0000) 2023-07-14T18:13:36.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:35 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:35.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:36.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:35 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:35.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:36.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:35 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:35.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:36.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:35 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:35.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:36.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:35 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:35.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:36.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:35 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:35.762+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:36.187 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:35.846+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.294333+0000 front 2023-07-14T18:12:41.294292+0000 (oldest deadline 2023-07-14T18:13:05.394171+0000) 2023-07-14T18:13:36.187 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:35.846+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.295175+0000 front 2023-07-14T18:12:51.295198+0000 (oldest deadline 2023-07-14T18:13:12.395318+0000) 2023-07-14T18:13:36.187 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:35.846+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.296365+0000 front 2023-07-14T18:12:58.296257+0000 (oldest deadline 2023-07-14T18:13:22.396231+0000) 2023-07-14T18:13:36.188 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:35.846+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.497136+0000 front 2023-07-14T18:13:06.497277+0000 (oldest deadline 2023-07-14T18:13:27.597080+0000) 2023-07-14T18:13:36.188 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:35 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8[146080]: 2023-07-14T18:13:35.846+0000 7fcb6dd4f700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.297895+0000 front 2023-07-14T18:13:12.297876+0000 (oldest deadline 2023-07-14T18:13:35.798072+0000) 2023-07-14T18:13:36.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:36 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:36.236+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:36.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:36 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:36.236+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:36.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:36 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:36.236+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:36.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:36 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:36.236+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:36.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:36 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:36.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:36.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:36 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:36.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:36.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:36 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:36.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:36.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:36 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:36.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:36.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:36 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:36.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:36.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:36 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:36.482+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.595922+0000 front 2023-07-14T18:13:12.596059+0000 (oldest deadline 2023-07-14T18:13:34.895982+0000) 2023-07-14T18:13:37.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:36 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:36.776+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:37.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:36 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:36.776+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:37.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:36 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:36.776+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:37.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:36 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:36.776+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:37.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:36 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:36.776+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:37.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:36 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:36.776+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:37.187 INFO:journalctl@ceph.osd.8.smithi187.stdout:Jul 14 18:13:36 smithi187 bash[158343]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-8 2023-07-14T18:13:37.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:37 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:37.194+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:37.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:37 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:37.194+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:37.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:37 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:37.194+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:37.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:37 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:37.194+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:37.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:37 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:37.526+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:37.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:37 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:37.526+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:37.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:37 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:37.526+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:37.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:37 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:37.526+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:37.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:37 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:37.526+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:37.861 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:37 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:37.526+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.595922+0000 front 2023-07-14T18:13:12.596059+0000 (oldest deadline 2023-07-14T18:13:34.895982+0000) 2023-07-14T18:13:38.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:37 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:37.816+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:38.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:37 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:37.816+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:38.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:37 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:37.816+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:38.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:37 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:37.816+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:38.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:37 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:37.816+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:38.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:37 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:37.816+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:38.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:38 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:38.155+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:38.437 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:38 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:38.155+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:38.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:38 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:38.156+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:38.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:38 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:38.156+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:38.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:38 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:38.156+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:38.445 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.8.service' 2023-07-14T18:13:38.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:13:38.487 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-14T18:13:38.487 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-14T18:13:38.488 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.9 2023-07-14T18:13:38.835 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:38.478+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:38.836 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:38.478+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:38.836 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:38.478+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:38.836 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:38.478+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:38.837 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:38.478+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:38.837 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:38.478+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.595922+0000 front 2023-07-14T18:13:12.596059+0000 (oldest deadline 2023-07-14T18:13:34.895982+0000) 2023-07-14T18:13:38.837 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 systemd[1]: Stopping Ceph osd.9 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:13:39.109 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:38.794+0000 7fae38d8e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:13:39.110 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:38.794+0000 7fae38d8e700 -1 osd.9 93 *** Got signal Terminated *** 2023-07-14T18:13:39.110 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:38 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:38.794+0000 7fae38d8e700 -1 osd.9 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:13:39.374 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:38 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:38.846+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:39.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:38 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:38.846+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:39.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:38 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:38.846+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:39.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:38 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:38.846+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:39.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:38 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:38.846+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:39.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:38 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:38.846+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:39.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:39 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:39.191+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:39.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:39 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:39.191+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:39.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:39 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:39.191+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:39.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:39 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:39.191+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:39.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:39 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:39.191+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:39.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:39 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:39.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:39.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:39 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:39.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:39.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:39 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:39.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:39.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:39 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:39.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:39.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:39 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:39.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:39.861 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:39 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:39.521+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.595922+0000 front 2023-07-14T18:13:12.596059+0000 (oldest deadline 2023-07-14T18:13:34.895982+0000) 2023-07-14T18:13:40.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:39 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:39.824+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:40.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:39 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:39.824+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:40.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:39 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:39.824+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:40.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:39 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:39.824+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:40.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:39 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:39.824+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:40.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:39 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:39.824+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:40.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:40 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:40.239+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:40.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:40 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:40.239+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:40.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:40 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:40.239+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:40.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:40 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:40.239+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:40.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:40 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:40.239+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:40.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:40 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:40.499+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:40.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:40 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:40.499+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:40.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:40 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:40.499+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:40.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:40 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:40.499+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:40.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:40 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:40.499+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:40.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:40 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:40.499+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.595922+0000 front 2023-07-14T18:13:12.596059+0000 (oldest deadline 2023-07-14T18:13:34.895982+0000) 2023-07-14T18:13:41.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:40 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:40.811+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:41.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:40 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:40.811+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:41.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:40 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:40.811+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:41.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:40 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:40.811+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:41.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:40 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:40.811+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:41.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:40 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:40.811+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:41.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:41 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:41.194+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:41.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:41 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:41.194+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:41.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:41 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:41.194+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:41.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:41 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:41.194+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:41.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:41 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:41.194+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:41.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:41 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:41.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:41.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:41 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:41.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:41.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:41 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:41.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:41.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:41 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:41.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:41.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:41 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:41.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:41.861 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:41 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:41.453+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.595922+0000 front 2023-07-14T18:13:12.596059+0000 (oldest deadline 2023-07-14T18:13:34.895982+0000) 2023-07-14T18:13:42.124 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:41 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:41.818+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:42.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:41 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:41.818+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:42.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:41 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:41.818+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:42.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:41 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:41.818+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:42.125 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:41 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:41.818+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:42.126 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:41 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:41.818+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:42.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:42 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:42.232+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:42.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:42 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:42.232+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:42.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:42 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:42.233+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:42.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:42 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:42.233+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:42.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:42 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:42.233+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:42.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:42 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:42.477+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:42.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:42 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:42.477+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:42.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:42 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:42.477+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:42.861 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:42 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:42.477+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:42.861 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:42 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:42.477+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:42.862 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:42 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:42.477+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.595922+0000 front 2023-07-14T18:13:12.596059+0000 (oldest deadline 2023-07-14T18:13:34.895982+0000) 2023-07-14T18:13:43.374 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:42 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:42.856+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:43.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:42 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:42.856+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:43.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:42 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:42.856+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:43.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:42 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:42.856+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:43.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:42 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:42.856+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:43.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:42 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:42.856+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:43.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:43 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:43.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:43.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:43 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:43.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:43.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:43 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:43.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:43.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:43 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:43.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:43.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:43 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:43.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:43.859 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:43 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:43.483+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:34.789626+0000 front 2023-07-14T18:12:34.789646+0000 (oldest deadline 2023-07-14T18:12:57.089784+0000) 2023-07-14T18:13:43.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:43 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:43.483+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:41.190850+0000 front 2023-07-14T18:12:41.191014+0000 (oldest deadline 2023-07-14T18:13:05.890916+0000) 2023-07-14T18:13:43.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:43 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:43.483+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.792199+0000 front 2023-07-14T18:12:51.792146+0000 (oldest deadline 2023-07-14T18:13:13.492024+0000) 2023-07-14T18:13:43.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:43 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:43.483+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:56.392814+0000 front 2023-07-14T18:12:56.392846+0000 (oldest deadline 2023-07-14T18:13:21.092921+0000) 2023-07-14T18:13:43.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:43 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:43.483+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:06.795199+0000 front 2023-07-14T18:13:06.795171+0000 (oldest deadline 2023-07-14T18:13:27.294861+0000) 2023-07-14T18:13:43.860 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:43 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9[156892]: 2023-07-14T18:13:43.483+0000 7fae33381700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.595922+0000 front 2023-07-14T18:13:12.596059+0000 (oldest deadline 2023-07-14T18:13:34.895982+0000) 2023-07-14T18:13:44.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:43 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:43.898+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:44.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:43 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:43.898+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:44.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:43 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:43.898+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:44.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:43 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:43.898+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:44.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:43 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:43.898+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:44.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:43 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:43.898+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:44.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:43 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:43.898+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:19.165888+0000 front 2023-07-14T18:13:19.166068+0000 (oldest deadline 2023-07-14T18:13:43.865752+0000) 2023-07-14T18:13:44.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:44 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:44.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:44.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:44 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:44.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:44.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:44 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:44.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:44.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:44 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:44.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:44.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:44 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:44.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:44.727 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:44 smithi115 bash[180292]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-9 2023-07-14T18:13:45.374 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:44 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:44.872+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:45.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:44 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:44.872+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:45.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:44 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:44.872+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:45.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:44 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:44.872+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:45.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:44 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:44.872+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:45.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:44 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:44.872+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:45.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:44 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:44.872+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:19.165888+0000 front 2023-07-14T18:13:19.166068+0000 (oldest deadline 2023-07-14T18:13:43.865752+0000) 2023-07-14T18:13:45.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:45 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:45.236+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:45.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:45 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:45.236+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:45.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:45 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:45.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:45.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:45 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:45.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:45.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:45 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:45.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:46.088 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.9.service' 2023-07-14T18:13:46.118 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:46 smithi115 systemd[1]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.9.service: Succeeded. 2023-07-14T18:13:46.118 INFO:journalctl@ceph.osd.9.smithi115.stdout:Jul 14 18:13:46 smithi115 systemd[1]: Stopped Ceph osd.9 for a39f385c-2270-11ee-9b33-001a4aab830c. 2023-07-14T18:13:46.129 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:13:46.129 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-14T18:13:46.130 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-14T18:13:46.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.10 2023-07-14T18:13:46.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:45 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:45.894+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:46.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:45 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:45.894+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:46.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:45 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:45.894+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:46.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:45 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:45.894+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:46.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:45 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:45.894+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:46.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:45 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:45.894+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:46.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:45 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:45.894+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:19.165888+0000 front 2023-07-14T18:13:19.166068+0000 (oldest deadline 2023-07-14T18:13:43.865752+0000) 2023-07-14T18:13:46.377 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 systemd[1]: Stopping Ceph osd.10 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:13:46.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:46 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:46.204+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:46.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:46 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:46.204+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:46.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:46 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:46.204+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:46.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:46 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:46.204+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:46.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:46 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:46.204+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:46.874 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.434+0000 7f68371f8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:13:46.875 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.434+0000 7f68371f8700 -1 osd.10 93 *** Got signal Terminated *** 2023-07-14T18:13:46.875 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.434+0000 7f68371f8700 -1 osd.10 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:13:47.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.885+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:47.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.885+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:47.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.885+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:47.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.885+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:47.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.885+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:47.377 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.885+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:47.377 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:46 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:46.885+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:19.165888+0000 front 2023-07-14T18:13:19.166068+0000 (oldest deadline 2023-07-14T18:13:43.865752+0000) 2023-07-14T18:13:47.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:47 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:47.171+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:47.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:47 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:47.171+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:47.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:47 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:47.171+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:47.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:47 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:47.171+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:47.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:47 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:47.171+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:47.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:47 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:47.171+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:48.374 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:47 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:47.858+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:48.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:47 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:47.858+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:48.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:47 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:47.858+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:48.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:47 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:47.858+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:48.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:47 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:47.858+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:48.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:47 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:47.858+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:48.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:47 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:47.858+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:19.165888+0000 front 2023-07-14T18:13:19.166068+0000 (oldest deadline 2023-07-14T18:13:43.865752+0000) 2023-07-14T18:13:48.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:48 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:48.175+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:48.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:48 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:48.175+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:48.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:48 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:48.175+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:48.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:48 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:48.175+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:48.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:48 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:48.175+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:48.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:48 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:48.175+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:49.374 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:48 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:48.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:49.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:48 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:48.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:49.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:48 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:48.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:49.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:48 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:48.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:49.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:48 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:48.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:49.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:48 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:48.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:49.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:48 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:48.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:19.165888+0000 front 2023-07-14T18:13:19.166068+0000 (oldest deadline 2023-07-14T18:13:43.865752+0000) 2023-07-14T18:13:49.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:49 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:49.213+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:49.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:49 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:49.213+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:49.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:49 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:49.213+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:49.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:49 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:49.213+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:49.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:49 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:49.213+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:49.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:49 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:49.213+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:50.374 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:49 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:49.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:50.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:49 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:49.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:50.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:49 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:49.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:50.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:49 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:49.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:50.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:49 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:49.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:50.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:49 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:49.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:50.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:49 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:49.901+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:19.165888+0000 front 2023-07-14T18:13:19.166068+0000 (oldest deadline 2023-07-14T18:13:43.865752+0000) 2023-07-14T18:13:50.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:50 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:50.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:50.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:50 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:50.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:50.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:50 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:50.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:50.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:50 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:50.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:50.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:50 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:50.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:50.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:50 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:50.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:51.374 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:50 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:50.854+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:33.058534+0000 front 2023-07-14T18:12:33.058498+0000 (oldest deadline 2023-07-14T18:12:58.958644+0000) 2023-07-14T18:13:51.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:50 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:50.854+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:38.958919+0000 front 2023-07-14T18:12:38.958982+0000 (oldest deadline 2023-07-14T18:13:04.859064+0000) 2023-07-14T18:13:51.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:50 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:50.854+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.261070+0000 front 2023-07-14T18:12:51.261166+0000 (oldest deadline 2023-07-14T18:13:14.161180+0000) 2023-07-14T18:13:51.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:50 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:50.854+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6814 osd.3 since back 2023-07-14T18:12:58.262349+0000 front 2023-07-14T18:12:58.262285+0000 (oldest deadline 2023-07-14T18:13:20.562285+0000) 2023-07-14T18:13:51.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:50 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:50.854+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:05.664065+0000 front 2023-07-14T18:13:05.664104+0000 (oldest deadline 2023-07-14T18:13:28.564258+0000) 2023-07-14T18:13:51.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:50 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:50.854+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:08.564975+0000 front 2023-07-14T18:13:08.564855+0000 (oldest deadline 2023-07-14T18:13:34.464706+0000) 2023-07-14T18:13:51.376 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:50 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10[150254]: 2023-07-14T18:13:50.854+0000 7f68337ef700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:19.165888+0000 front 2023-07-14T18:13:19.166068+0000 (oldest deadline 2023-07-14T18:13:43.865752+0000) 2023-07-14T18:13:51.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:51 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:51.218+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:51.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:51 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:51.218+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:51.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:51 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:51.218+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:51.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:51 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:51.218+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:51.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:51 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:51.218+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:51.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:51 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:51.218+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:52.375 INFO:journalctl@ceph.osd.10.smithi150.stdout:Jul 14 18:13:51 smithi150 bash[157234]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-10 2023-07-14T18:13:52.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:52 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:52.223+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:52.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:52 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:52.223+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:52.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:52 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:52.223+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:52.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:52 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:52.223+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:52.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:52 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:52.223+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:52.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:52 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:52.223+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:53.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.10.service' 2023-07-14T18:13:53.233 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:13:53.233 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-14T18:13:53.234 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-14T18:13:53.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.11 2023-07-14T18:13:53.596 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:53.208+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:53.596 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:53.208+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:53.597 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:53.208+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:53.597 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:53.208+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:53.597 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:53.208+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:53.597 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:53.208+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:53.598 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 systemd[1]: Stopping Ceph osd.11 for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:13:53.937 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:53.553+0000 7fef280f6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:13:53.937 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:53.553+0000 7fef280f6700 -1 osd.11 93 *** Got signal Terminated *** 2023-07-14T18:13:53.938 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:53 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:53.553+0000 7fef280f6700 -1 osd.11 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:13:54.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:54 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:54.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:54.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:54 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:54.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:54.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:54 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:54.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:54.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:54 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:54.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:54.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:54 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:54.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:54.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:54 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:54.193+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:55.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:55 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:55.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:55.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:55 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:55.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:55.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:55 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:55.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:55.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:55 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:55.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:55.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:55 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:55.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:55.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:55 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:55.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:55.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:55 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:55.206+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2023-07-14T18:13:28.994698+0000 front 2023-07-14T18:13:28.994728+0000 (oldest deadline 2023-07-14T18:13:54.894786+0000) 2023-07-14T18:13:56.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:56 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:56.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:56.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:56 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:56.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:56.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:56 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:56.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:56.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:56 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:56.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:56.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:56 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:56.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:56.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:56 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:56.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:56.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:56 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:56.237+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2023-07-14T18:13:28.994698+0000 front 2023-07-14T18:13:28.994728+0000 (oldest deadline 2023-07-14T18:13:54.894786+0000) 2023-07-14T18:13:57.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:57 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:57.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:57.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:57 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:57.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:57.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:57 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:57.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:57.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:57 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:57.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:57.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:57 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:57.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:57.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:57 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:57.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:57.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:57 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:57.217+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2023-07-14T18:13:28.994698+0000 front 2023-07-14T18:13:28.994728+0000 (oldest deadline 2023-07-14T18:13:54.894786+0000) 2023-07-14T18:13:58.687 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:58.221+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2023-07-14T18:12:35.385898+0000 front 2023-07-14T18:12:35.385727+0000 (oldest deadline 2023-07-14T18:12:58.885867+0000) 2023-07-14T18:13:58.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:58.221+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6804 osd.1 since back 2023-07-14T18:12:43.586993+0000 front 2023-07-14T18:12:43.587063+0000 (oldest deadline 2023-07-14T18:13:05.887038+0000) 2023-07-14T18:13:58.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:58.221+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6804 osd.2 since back 2023-07-14T18:12:51.787935+0000 front 2023-07-14T18:12:51.787898+0000 (oldest deadline 2023-07-14T18:13:12.288026+0000) 2023-07-14T18:13:58.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:58.221+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2023-07-14T18:13:04.991684+0000 front 2023-07-14T18:13:04.991677+0000 (oldest deadline 2023-07-14T18:13:29.091627+0000) 2023-07-14T18:13:58.688 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:58.221+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.187:6812 osd.5 since back 2023-07-14T18:13:12.592607+0000 front 2023-07-14T18:13:12.592653+0000 (oldest deadline 2023-07-14T18:13:37.892899+0000) 2023-07-14T18:13:58.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:58.221+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.115:6822 osd.6 since back 2023-07-14T18:13:21.394022+0000 front 2023-07-14T18:13:21.393994+0000 (oldest deadline 2023-07-14T18:13:46.693927+0000) 2023-07-14T18:13:58.689 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:58 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11[151653]: 2023-07-14T18:13:58.221+0000 7fef206e5700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.150:6820 osd.7 since back 2023-07-14T18:13:28.994698+0000 front 2023-07-14T18:13:28.994728+0000 (oldest deadline 2023-07-14T18:13:54.894786+0000) 2023-07-14T18:13:59.374 INFO:journalctl@ceph.osd.11.smithi187.stdout:Jul 14 18:13:59 smithi187 bash[158616]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-osd-11 2023-07-14T18:14:00.739 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@osd.11.service' 2023-07-14T18:14:00.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:00.780 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-14T18:14:00.780 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-14T18:14:00.781 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.a 2023-07-14T18:14:01.110 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:14:00 smithi115 systemd[1]: Stopping Ceph mds.a for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:01.610 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:14:01 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-a[162178]: 2023-07-14T18:14:01.095+0000 7f838c645700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:01.610 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:14:01 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-a[162178]: 2023-07-14T18:14:01.095+0000 7f838c645700 -1 mds.a *** got signal Terminated *** 2023-07-14T18:14:05.609 INFO:journalctl@ceph.mds.a.smithi115.stdout:Jul 14 18:14:05 smithi115 bash[180564]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-a 2023-07-14T18:14:05.627 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.a.service' 2023-07-14T18:14:05.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:05.664 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-14T18:14:05.664 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-14T18:14:05.664 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.d 2023-07-14T18:14:06.054 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:14:05 smithi115 systemd[1]: Stopping Ceph mds.d for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:06.359 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:14:06 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-d[161176]: 2023-07-14T18:14:06.011+0000 7fcc7cfb7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:06.360 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:14:06 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-d[161176]: 2023-07-14T18:14:06.011+0000 7fcc7cfb7700 -1 mds.d *** got signal Terminated *** 2023-07-14T18:14:10.588 INFO:journalctl@ceph.mds.d.smithi115.stdout:Jul 14 18:14:10 smithi115 bash[180714]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-d 2023-07-14T18:14:10.594 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.d.service' 2023-07-14T18:14:10.628 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:10.629 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-14T18:14:10.629 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-14T18:14:10.630 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.g 2023-07-14T18:14:10.860 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:14:10 smithi115 systemd[1]: Stopping Ceph mds.g for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:11.360 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:14:10 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-g[161790]: 2023-07-14T18:14:10.967+0000 7fdeb6ce8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:11.360 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:14:10 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-g[161790]: 2023-07-14T18:14:10.967+0000 7fdeb6ce8700 -1 mds.g *** got signal Terminated *** 2023-07-14T18:14:15.515 INFO:journalctl@ceph.mds.g.smithi115.stdout:Jul 14 18:14:15 smithi115 bash[180861]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-g 2023-07-14T18:14:15.521 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.g.service' 2023-07-14T18:14:15.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:15.555 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-14T18:14:15.556 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-14T18:14:15.556 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.j 2023-07-14T18:14:15.860 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:14:15 smithi115 systemd[1]: Stopping Ceph mds.j for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:16.360 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:14:15 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-j[160815]: 2023-07-14T18:14:15.883+0000 7fc34848f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:16.360 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:14:15 smithi115 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-j[160815]: 2023-07-14T18:14:15.883+0000 7fc34848f700 -1 mds.j *** got signal Terminated *** 2023-07-14T18:14:20.360 INFO:journalctl@ceph.mds.j.smithi115.stdout:Jul 14 18:14:20 smithi115 bash[181006]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-j 2023-07-14T18:14:20.546 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.j.service' 2023-07-14T18:14:20.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:20.579 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-14T18:14:20.580 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-14T18:14:20.580 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.b 2023-07-14T18:14:20.875 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:14:20 smithi150 systemd[1]: Stopping Ceph mds.b for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:21.375 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:14:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-b[154137]: 2023-07-14T18:14:20.880+0000 7f46f37f9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:21.375 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:14:20 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-b[154137]: 2023-07-14T18:14:20.880+0000 7f46f37f9700 -1 mds.b *** got signal Terminated *** 2023-07-14T18:14:25.125 INFO:journalctl@ceph.mds.b.smithi150.stdout:Jul 14 18:14:24 smithi150 bash[157509]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-b 2023-07-14T18:14:25.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.b.service' 2023-07-14T18:14:25.199 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:25.200 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-14T18:14:25.200 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-14T18:14:25.201 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.e 2023-07-14T18:14:25.572 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:14:25 smithi150 systemd[1]: Stopping Ceph mds.e for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:25.875 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:14:25 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-e[153341]: 2023-07-14T18:14:25.528+0000 7f59a18a1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:25.876 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:14:25 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-e[153341]: 2023-07-14T18:14:25.528+0000 7f59a18a1700 -1 mds.e *** got signal Terminated *** 2023-07-14T18:14:29.375 INFO:journalctl@ceph.mds.e.smithi150.stdout:Jul 14 18:14:29 smithi150 bash[157659]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-e 2023-07-14T18:14:29.435 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.e.service' 2023-07-14T18:14:29.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:29.469 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-14T18:14:29.469 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-14T18:14:29.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.h 2023-07-14T18:14:29.875 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:14:29 smithi150 systemd[1]: Stopping Ceph mds.h for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:30.375 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:14:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-h[153682]: 2023-07-14T18:14:29.845+0000 7f9104758700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:30.376 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:14:29 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-h[153682]: 2023-07-14T18:14:29.845+0000 7f9104758700 -1 mds.h *** got signal Terminated *** 2023-07-14T18:14:33.125 INFO:journalctl@ceph.mds.h.smithi150.stdout:Jul 14 18:14:32 smithi150 bash[157806]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-h 2023-07-14T18:14:33.202 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.h.service' 2023-07-14T18:14:33.236 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:33.236 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-14T18:14:33.237 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-14T18:14:33.237 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.k 2023-07-14T18:14:33.625 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:14:33 smithi150 systemd[1]: Stopping Ceph mds.k for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:34.125 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:14:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-k[152991]: 2023-07-14T18:14:33.625+0000 7f1e2184e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:34.126 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:14:33 smithi150 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-k[152991]: 2023-07-14T18:14:33.625+0000 7f1e2184e700 -1 mds.k *** got signal Terminated *** 2023-07-14T18:14:38.297 INFO:journalctl@ceph.mds.k.smithi150.stdout:Jul 14 18:14:37 smithi150 bash[157943]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-k 2023-07-14T18:14:38.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.k.service' 2023-07-14T18:14:38.335 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:38.335 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-14T18:14:38.336 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-14T18:14:38.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.c 2023-07-14T18:14:38.688 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:14:38 smithi187 systemd[1]: Stopping Ceph mds.c for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:39.188 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:14:38 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-c[155141]: 2023-07-14T18:14:38.654+0000 7ffada47e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:39.189 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:14:38 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-c[155141]: 2023-07-14T18:14:38.654+0000 7ffada47e700 -1 mds.c *** got signal Terminated *** 2023-07-14T18:14:41.055 INFO:journalctl@ceph.mds.c.smithi187.stdout:Jul 14 18:14:40 smithi187 bash[158886]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-c 2023-07-14T18:14:41.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.c.service' 2023-07-14T18:14:41.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:41.283 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-14T18:14:41.283 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-14T18:14:41.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.f 2023-07-14T18:14:41.655 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:14:41 smithi187 systemd[1]: Stopping Ceph mds.f for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:41.938 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:14:41 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-f[154799]: 2023-07-14T18:14:41.611+0000 7f527639b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:41.938 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:14:41 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-f[154799]: 2023-07-14T18:14:41.611+0000 7f527639b700 -1 mds.f *** got signal Terminated *** 2023-07-14T18:14:45.188 INFO:journalctl@ceph.mds.f.smithi187.stdout:Jul 14 18:14:44 smithi187 bash[159027]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-f 2023-07-14T18:14:45.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.f.service' 2023-07-14T18:14:45.308 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:45.309 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-14T18:14:45.309 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-14T18:14:45.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.i 2023-07-14T18:14:45.679 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:14:45 smithi187 systemd[1]: Stopping Ceph mds.i for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:45.938 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:14:45 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-i[155482]: 2023-07-14T18:14:45.634+0000 7f1e12893700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:45.938 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:14:45 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-i[155482]: 2023-07-14T18:14:45.635+0000 7f1e12893700 -1 mds.i *** got signal Terminated *** 2023-07-14T18:14:49.052 INFO:journalctl@ceph.mds.i.smithi187.stdout:Jul 14 18:14:48 smithi187 bash[159177]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-i 2023-07-14T18:14:49.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.i.service' 2023-07-14T18:14:49.272 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:49.273 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-14T18:14:49.273 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-14T18:14:49.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.l 2023-07-14T18:14:49.653 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:14:49 smithi187 systemd[1]: Stopping Ceph mds.l for a39f385c-2270-11ee-9b33-001a4aab830c... 2023-07-14T18:14:49.938 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:14:49 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-l[154461]: 2023-07-14T18:14:49.608+0000 7f781b8ca700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:14:49.938 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:14:49 smithi187 ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-l[154461]: 2023-07-14T18:14:49.608+0000 7f781b8ca700 -1 mds.l *** got signal Terminated *** 2023-07-14T18:14:52.328 INFO:journalctl@ceph.mds.l.smithi187.stdout:Jul 14 18:14:52 smithi187 bash[159314]: ceph-a39f385c-2270-11ee-9b33-001a4aab830c-mds-l 2023-07-14T18:14:52.445 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a39f385c-2270-11ee-9b33-001a4aab830c@mds.l.service' 2023-07-14T18:14:52.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:14:52.480 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-14T18:14:52.481 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a39f385c-2270-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T18:14:52.785 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:14:54.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a39f385c-2270-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T18:14:54.633 INFO:teuthology.orchestra.run.smithi150.stdout:Deleting cluster with fsid: a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:14:56.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a39f385c-2270-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T18:14:56.472 INFO:teuthology.orchestra.run.smithi187.stdout:Deleting cluster with fsid: a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:14:58.044 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:14:58.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:14:58.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:14:58.146 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-14T18:14:58.148 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/remote/smithi115/crash 2023-07-14T18:14:58.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T18:14:58.205 INFO:teuthology.orchestra.run.smithi115.stderr:tar: /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T18:14:58.206 INFO:teuthology.orchestra.run.smithi115.stderr:tar: Error is not recoverable: exiting now 2023-07-14T18:14:58.207 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/remote/smithi150/crash 2023-07-14T18:14:58.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T18:14:58.258 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T18:14:58.259 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-07-14T18:14:58.260 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/remote/smithi187/crash 2023-07-14T18:14:58.261 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T18:14:58.299 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T18:14:58.300 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2023-07-14T18:14:58.300 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-14T18:14:58.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-14T18:14:58.335 INFO:teuthology.orchestra.run.smithi115.stderr:grep: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph.log: No such file or directory 2023-07-14T18:14:58.337 INFO:tasks.cephadm:Compressing logs... 2023-07-14T18:14:58.337 DEBUG:teuthology.orchestra.run.smithi115:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T18:14:58.379 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T18:14:58.381 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T18:14:58.405 INFO:teuthology.orchestra.run.smithi115.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.0.log 2023-07-14T18:14:58.406 INFO:teuthology.orchestra.run.smithi115.stderr:: No such file or directory 2023-07-14T18:14:58.406 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.g.log 2023-07-14T18:14:58.407 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.a.log 2023-07-14T18:14:58.407 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.0.log: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.6.log 2023-07-14T18:14:58.407 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.3.log 2023-07-14T18:14:58.408 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.d.log 2023-07-14T18:14:58.409 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.3.log: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.a.log.gz 2023-07-14T18:14:58.409 INFO:teuthology.orchestra.run.smithi115.stderr: 94.6% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.g.log.gz 2023-07-14T18:14:58.410 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mgr.x.log 2023-07-14T18:14:58.410 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.9.log 2023-07-14T18:14:58.410 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.5.log 2023-07-14T18:14:58.410 INFO:teuthology.orchestra.run.smithi187.stderr:: No such file or directory 2023-07-14T18:14:58.411 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.2.log 2023-07-14T18:14:58.411 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.11.log 2023-07-14T18:14:58.411 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.5.log: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mgr.z.log 2023-07-14T18:14:58.411 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.11.log: -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.8.log 2023-07-14T18:14:58.411 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.f.log 2023-07-14T18:14:58.412 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.c.log 2023-07-14T18:14:58.413 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mgr.x.log: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.9.log: 94.5% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.d.log.gz 2023-07-14T18:14:58.413 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T18:14:58.413 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.j.log 2023-07-14T18:14:58.414 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.10.log 2023-07-14T18:14:58.414 INFO:teuthology.orchestra.run.smithi150.stderr:: No such file or directory 2023-07-14T18:14:58.414 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.h.log 2023-07-14T18:14:58.415 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.e.log 2023-07-14T18:14:58.415 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.4.log 2023-07-14T18:14:58.415 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.1.log 2023-07-14T18:14:58.415 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.k.log 2023-07-14T18:14:58.416 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T18:14:58.416 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.k.log: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-volume.log: 94.7% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.k.log.gz 2023-07-14T18:14:58.417 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-volume.log: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.j.log: 94.7% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.j.log.gz 2023-07-14T18:14:58.417 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mon.a.log 2023-07-14T18:14:58.418 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T18:14:58.419 INFO:teuthology.orchestra.run.smithi115.stderr: 91.6% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T18:14:58.420 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2023-07-14T18:14:58.420 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.f.log: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.c.log: 91.6% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mgr.z.log.gz 2023-07-14T18:14:58.420 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.i.log 2023-07-14T18:14:58.420 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mon.c.log 2023-07-14T18:14:58.421 INFO:teuthology.orchestra.run.smithi150.stderr: 91.5% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T18:14:58.421 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.7.log 2023-07-14T18:14:58.421 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mon.b.log 2023-07-14T18:14:58.423 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.admin.log: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2023-07-14T18:14:58.423 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2023-07-14T18:14:58.424 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.7.log: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.b.log 2023-07-14T18:14:58.425 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/ceph-client.0.log: 91.8% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2023-07-14T18:14:58.426 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.i.log: /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T18:14:58.426 INFO:teuthology.orchestra.run.smithi187.stderr: 92.8% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.c.log.gz 2023-07-14T18:14:58.426 INFO:teuthology.orchestra.run.smithi115.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T18:14:58.428 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-volume.log: 91.3% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.f.log.gz 2023-07-14T18:14:58.428 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.l.log 2023-07-14T18:14:58.431 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.l.log: 94.8% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.l.log.gz 2023-07-14T18:14:58.431 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T18:14:58.432 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.b.log: 92.7% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.e.log.gz 2023-07-14T18:14:58.432 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mgr.y.log 2023-07-14T18:14:58.432 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mgr.y.log: 92.5% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.h.log.gz 2023-07-14T18:14:58.432 INFO:teuthology.orchestra.run.smithi150.stderr: 91.5% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mgr.y.log.gz 2023-07-14T18:14:58.432 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T18:14:58.433 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/cephadm.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T18:14:58.434 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T18:14:58.434 INFO:teuthology.orchestra.run.smithi187.stderr: 91.5% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T18:14:58.442 INFO:teuthology.orchestra.run.smithi187.stderr: 91.3% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.i.log.gz 2023-07-14T18:14:58.518 INFO:teuthology.orchestra.run.smithi150.stderr: 92.6% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mds.b.log.gz 2023-07-14T18:14:58.547 INFO:teuthology.orchestra.run.smithi150.stderr: 91.9% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mon.b.log.gz 2023-07-14T18:14:58.552 INFO:teuthology.orchestra.run.smithi187.stderr: 91.9% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mon.c.log.gz 2023-07-14T18:14:58.555 INFO:teuthology.orchestra.run.smithi115.stderr: 89.1% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mgr.x.log.gz 2023-07-14T18:14:58.637 INFO:teuthology.orchestra.run.smithi187.stderr: 92.5% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.8.log.gz 2023-07-14T18:14:58.645 INFO:teuthology.orchestra.run.smithi115.stderr: 92.4% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.6.log.gz 2023-07-14T18:14:58.646 INFO:teuthology.orchestra.run.smithi187.stderr: 92.9% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.11.log.gz 2023-07-14T18:14:58.661 INFO:teuthology.orchestra.run.smithi115.stderr: 92.6% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.9.log.gz 2023-07-14T18:14:58.662 INFO:teuthology.orchestra.run.smithi115.stderr: 92.4% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.3.log.gz 2023-07-14T18:14:58.675 INFO:teuthology.orchestra.run.smithi150.stderr: 92.5% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.7.log.gz 2023-07-14T18:14:58.686 INFO:teuthology.orchestra.run.smithi150.stderr: 92.5% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.10.log.gz 2023-07-14T18:14:58.711 INFO:teuthology.orchestra.run.smithi187.stderr: 92.3% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.2.log.gz 2023-07-14T18:14:58.718 INFO:teuthology.orchestra.run.smithi150.stderr: 92.4% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.4.log.gz 2023-07-14T18:14:58.754 INFO:teuthology.orchestra.run.smithi187.stderr: 92.4% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.5.log.gz 2023-07-14T18:14:58.755 INFO:teuthology.orchestra.run.smithi187.stderr: 2023-07-14T18:14:58.755 INFO:teuthology.orchestra.run.smithi187.stderr:real 0m0.358s 2023-07-14T18:14:58.756 INFO:teuthology.orchestra.run.smithi187.stderr:user 0m1.279s 2023-07-14T18:14:58.756 INFO:teuthology.orchestra.run.smithi187.stderr:sys 0m0.060s 2023-07-14T18:14:58.760 INFO:teuthology.orchestra.run.smithi115.stderr: 92.5% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.0.log.gz 2023-07-14T18:14:58.784 INFO:teuthology.orchestra.run.smithi150.stderr: 92.2% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-osd.1.log.gz 2023-07-14T18:14:58.786 INFO:teuthology.orchestra.run.smithi150.stderr: 2023-07-14T18:14:58.786 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m0.389s 2023-07-14T18:14:58.786 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m1.449s 2023-07-14T18:14:58.786 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.063s 2023-07-14T18:14:58.887 INFO:teuthology.orchestra.run.smithi115.stderr: 91.0% -- replaced with /var/log/ceph/a39f385c-2270-11ee-9b33-001a4aab830c/ceph-mon.a.log.gz 2023-07-14T18:14:58.889 INFO:teuthology.orchestra.run.smithi115.stderr: 2023-07-14T18:14:58.889 INFO:teuthology.orchestra.run.smithi115.stderr:real 0m0.495s 2023-07-14T18:14:58.889 INFO:teuthology.orchestra.run.smithi115.stderr:user 0m1.698s 2023-07-14T18:14:58.889 INFO:teuthology.orchestra.run.smithi115.stderr:sys 0m0.076s 2023-07-14T18:14:58.890 INFO:tasks.cephadm:Archiving logs... 2023-07-14T18:14:58.890 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/remote/smithi115/log 2023-07-14T18:14:58.891 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T18:14:59.163 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/remote/smithi150/log 2023-07-14T18:14:59.164 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T18:14:59.356 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/remote/smithi187/log 2023-07-14T18:14:59.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T18:14:59.530 INFO:tasks.cephadm:Removing cluster... 2023-07-14T18:14:59.531 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a39f385c-2270-11ee-9b33-001a4aab830c --force 2023-07-14T18:14:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:15:00.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a39f385c-2270-11ee-9b33-001a4aab830c --force 2023-07-14T18:15:00.579 INFO:teuthology.orchestra.run.smithi150.stdout:Deleting cluster with fsid: a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:15:01.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a39f385c-2270-11ee-9b33-001a4aab830c --force 2023-07-14T18:15:01.332 INFO:teuthology.orchestra.run.smithi187.stdout:Deleting cluster with fsid: a39f385c-2270-11ee-9b33-001a4aab830c 2023-07-14T18:15:01.791 INFO:tasks.cephadm:Removing cephadm ... 2023-07-14T18:15:01.792 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T18:15:01.813 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T18:15:01.834 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T18:15:01.851 INFO:tasks.cephadm:Teardown complete 2023-07-14T18:15:01.852 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-14T18:15:01.862 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-14T18:15:01.862 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T18:15:01.864 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T18:15:01.876 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T18:15:14.527 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-14T18:15:14.537 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-14T18:15:14.539 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:15:14.542 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:15:14.545 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:15:14.556 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2023-07-14T18:15:14.560 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:15:14.560 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2023-07-14T18:15:14.561 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +588us[ +588us] +/- 95ms 2023-07-14T18:15:14.561 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -3281us[-3320us] +/- 86ms 2023-07-14T18:15:14.561 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +2243us[+2243us] +/- 72ms 2023-07-14T18:15:14.561 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:15:14.562 INFO:teuthology.orchestra.run.smithi150.stderr:bash: ntpq: command not found 2023-07-14T18:15:14.564 INFO:teuthology.orchestra.run.smithi187.stderr:bash: ntpq: command not found 2023-07-14T18:15:14.566 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:15:14.567 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-07-14T18:15:14.567 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +620us[ +620us] +/- 95ms 2023-07-14T18:15:14.567 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -3269us[-3262us] +/- 86ms 2023-07-14T18:15:14.568 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +2269us[+2269us] +/- 72ms 2023-07-14T18:15:14.568 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:15:14.570 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:15:14.571 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2023-07-14T18:15:14.571 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 +626us[ +660us] +/- 95ms 2023-07-14T18:15:14.571 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -3251us[-3218us] +/- 86ms 2023-07-14T18:15:14.572 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +2275us[+2309us] +/- 72ms 2023-07-14T18:15:14.572 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:15:14.572 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-14T18:15:14.585 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-14T18:15:14.586 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-14T18:15:14.598 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T18:15:14.702 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T18:15:14.801 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T18:15:14.902 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:15:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:15:14.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:15:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:15:14.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:15:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:15:15.004 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-14T18:15:15.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-14T18:15:15.026 INFO:teuthology.task.internal:Duration was 1862.626587 seconds 2023-07-14T18:15:15.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-14T18:15:15.036 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-14T18:15:15.036 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T18:15:15.039 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T18:15:15.041 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T18:15:15.081 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:15:15.086 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:15:15.092 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:15:15.388 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-14T18:15:15.389 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2023-07-14T18:15:15.389 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T18:15:15.410 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2023-07-14T18:15:15.410 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T18:15:15.432 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2023-07-14T18:15:15.432 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T18:15:15.455 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-14T18:15:15.455 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T18:15:15.458 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T18:15:15.474 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T18:15:15.543 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-14T18:15:15.544 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T18:15:15.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T18:15:15.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T18:15:15.669 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-14T18:15:15.680 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-14T18:15:15.680 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T18:15:15.706 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T18:15:15.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T18:15:15.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-14T18:15:15.771 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:15:15.773 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:15:15.775 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:15:15.798 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2023-07-14T18:15:15.798 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2023-07-14T18:15:15.826 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2023-07-14T18:15:15.895 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:15:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:15:15.913 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:15:15.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:15:15.929 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:15:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:15:15.952 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-14T18:15:15.969 INFO:teuthology.task.internal:Transferring archived files... 2023-07-14T18:15:15.970 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/remote/smithi115 2023-07-14T18:15:15.971 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T18:15:16.011 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/remote/smithi150 2023-07-14T18:15:16.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T18:15:16.047 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990/remote/smithi187 2023-07-14T18:15:16.048 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T18:15:16.084 INFO:teuthology.task.internal:Removing archive directory... 2023-07-14T18:15:16.085 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T18:15:16.087 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T18:15:16.090 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T18:15:16.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-14T18:15:16.146 INFO:teuthology.task.internal:Not uploading archives. 2023-07-14T18:15:16.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-14T18:15:16.195 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-14T18:15:16.196 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T18:15:16.198 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T18:15:16.200 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T18:15:16.214 INFO:teuthology.orchestra.run.smithi115.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 18:15 /home/ubuntu/cephtest 2023-07-14T18:15:16.215 INFO:teuthology.orchestra.run.smithi150.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 18:15 /home/ubuntu/cephtest 2023-07-14T18:15:16.217 INFO:teuthology.orchestra.run.smithi187.stdout: 262170 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 18:15 /home/ubuntu/cephtest 2023-07-14T18:15:16.218 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-14T18:15:16.229 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-14T18:15:16.272 INFO:teuthology.nuke:Checking targets against current locks 2023-07-14T18:15:16.302 DEBUG:teuthology.nuke:shortname: smithi115 2023-07-14T18:15:16.302 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:15:16.333 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:35:29.870021', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T18:15:16.337 DEBUG:teuthology.nuke:shortname: smithi150 2023-07-14T18:15:16.338 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:15:16.363 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:35:29.868687', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T18:15:16.369 DEBUG:teuthology.nuke:shortname: smithi187 2023-07-14T18:15:16.370 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:15:16.397 INFO:teuthology.orchestra.console:Power off smithi115 2023-07-14T18:15:16.398 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T18:15:16.411 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337990', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:35:29.867096', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T18:15:16.423 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:16.444 INFO:teuthology.orchestra.console:Power off smithi150 2023-07-14T18:15:16.445 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T18:15:16.469 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:16.490 INFO:teuthology.orchestra.console:Power off smithi187 2023-07-14T18:15:16.491 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T18:15:16.518 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:20.461 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:20.505 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:20.542 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:24.576 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:24.620 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:24.658 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:24.788 INFO:teuthology.orchestra.console:Power off for smithi115 completed 2023-07-14T18:15:24.870 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2023-07-14T18:15:28.736 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:33.849 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:38.964 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:44.080 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:49.192 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:54.306 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:15:59.419 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:16:04.531 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:16:04.743 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-07-14T18:16:04.848 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} duration: 1862.626587152481 failure_reason: 'Command failed on smithi115 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs _nogroup random 0.01''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7a975a42ae3b42e2afb6beeb8740f302 status: fail success: false 2023-07-14T18:16:04.849 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T18:16:07.578 INFO:teuthology.run:FAIL